TA²N: Two-Stage Action Alignment Network for Few-Shot Action Recognition

Following standard few-shot action recognition setting, the dataset is divided into three distinct parts: training set $\mathcal{C}_{train}$, validation set $\mathcal{C}_{val}$, and test set $\mathcal{C}_{test}$. The training set contains sufficient labeled data for each class while there exist only a few labeled samples in the test set. The validation set is only used to evaluate the model during training. Moreover, there are no overlapping categories between these three sets. Generally, few-shot action recognition aims to train a classification network that can generalize well to novel classes in the test set. In the specific $N$-way $K$-shot few-shot learning setting, each episode contains a support set $\mathcal{S}$ sampled from the training set $\mathcal{C}_{train}$. It contains $N\times K$ samples from $N$ different classes where each class contains $K$ support samples. Then Q samples from each class are selected to form the query set $\mathcal{Q}$ which contains $N\times Q$ samples. The goal is to classify the $N\times Q$ query samples only with the $N\times K$ support samples.

For each input video, we follow the sampling strategy described in TSN (Wang et al. 2016), which divides a video into $T$ segments and then samples frames uniformly in each segment. Thus, each video is represented by a fixed-length frame sequence. Given the frame sequence $X=\{x_{1},x_{2},...,x_{T}\}$, a feature embedding network $f(\cdot)$ takes it as input and embeds the sequence $X$ into $T$ frame-level features $f_{X}=f(X)=\{f(x_{1}),f(x_{2}),\dots,f(x_{T})\}\in\mathbb{R}^{\rm C\times T\times H\times W}$. From this point, we will use $f_{s}$, $f_{q}$ to represent the video-level feature of the support sample and query sample, respectively.

To temporally align the action evolution among videos, similar motion patterns between videos should be aggregated to the same temporal location. We treat this as a global coordination task where the motion evolution of query video can be temporally rearranged to match the support ones. Accordingly, we model the motion evolution correlation $M\in\mathbb{R}^{T\times T}$ between support and query:

where $W_{k},W_{q}$ indicate linear projection layer, $dim$ is the dimension of feature $G(\hat{f})$, $G$ is the global average pooling in spatial dimensions whose output tensor shape is $C\times T\times 1\times 1$, i.e. the correlation is only calculated in the temporal dimension, $\operatorname{Softmax}$ limits the values in $M$ to $[0,1]$.

Then, we could temporally rearrange the query feature by calculating the matrix multiplication between the normalized motion correlation matrix $M$ and the query features:

Similar to $W_{k},W_{q}$, $W_{v}$ denotes linear projection layer. In order to keep feature-space consistent, this projection are also applied to support feature $\hat{f_{s}}$: $\tilde{f_{s}}=W_{v}\cdot G(\hat{f_{s}})$. This way, the same temporal location is in the consistent evolution and AEM in temporal aspect can be relieved. The illustration of TC is presented in Fig. 5.

Temporal coordination (TC) ensures the action evolves with same process along the duration time. However, the spatial variation of actor evolution, such as the positions of actors, also being critical for action recognition, which cannot be modeled by TC. Thus, we further devise a spatial manipulation to reduce the spatial variations in action evolution. On the basis of temporally well-aligned features, we aims to predict an spatial offset for each paired frames and then measure their similarity in the intersection area only, as shown in the top of Fig. 6.

Specifically, spatial coordination consists of two steps: light-weight offset prediction and offset mask generation. First, given the temporally well-aligned feature $\tilde{f_{s}}$ and $\tilde{f_{q}}$, they are feed into the offset predictor $S$ to predict spatial offset $O\in\mathbb{R}^{T\times 2}$ in $x$ and $y$ coordinates for all timestamps:

where $\operatorname{Cat}(\cdot)$ denotes concatenation along the channel. The detailed structure of $S$ is elaborated in our Supplementary Material. As shown in Fig.6, the predicted offset indicates the relative position vector of the action-specific region between query and support frames. Then, the intersection area can be located by its corresponding spatial offset.

To calculate the similarity in the intersection area in a differentiable way, for each frame, we use a generated offset mask $I$ to calculate the average feature in the intersection on each feature. Moreover, the value of the mask is 1 inside the intersection area and it gradually decreases to 0 on the edge. More details about mask generation are provided in the supplementary material. Then the masks are performed over the query and support features simultaneously as the weights of weighted average:

where $I_{o_{i}}$ is the generated mask for $i$-th frame.

In order to expand the exploration space of the offset predictor, we further add some perturbations on predicted offset and use the average of corresponding features of different perturbations.

Undergoing TC and SC, the action evolution misalignment among videos are dismissed in spatial-temporal aspect. The well-aligned paired features $\overline{f_{s}}$ and $\overline{f_{q}}$ are used in final distance measurement and classification as the prototypical network scheme (Snell, Swersky, and Zemel 2017).

The quantitative results are listed in Tab. 2. As shown in this table, our method outperforms the strong baseline ProtoNet (Snell, Swersky, and Zemel 2017) on all datasets and is competitive with state-of-the-art methods. OTAM is the current state-of-the-art method that focuses on temporal alignment (and the most related to ours). Compared to it, TA²N surpasses its performance by a significant margin on most settings and datasets, demonstrating the superiority of our proposed two-stage alignment framework for action alignment. Moreover, our TA²N aligns actions in the spatial-temporal aspect while OTAM only considers the temporal dimension.

Among four benchmarks, the TA²N gains the most significant improvement on the SSv2. This finding also concurs with our quantitative analysis on the state of misalignment in datasets, whereby SSv2 manifests the most serious misalignment problem. This further demonstrates the effectiveness of our alignment modules. Although the UCF101 possesses a relatively less serious condition, our TA²N could still improve its performance by learning a more consistent temporal feature.

We visualize the alignment results to illustrate the effectiveness of our proposed method, which are presented in Fig. 7. It can be observed that there exists a clear duration and spatial-temporal evolution misalignment between the support and query videos. It’s clear that the duration is well-aligned by the TTM, which filters the insignificant background frame noise. Besides, the spatial regions coordinated by SC focus on the common action-specific part between paired frames. For example, the region of hand (row 3 col 2 in Fig. 7) and the action-specific object ‘cup’ (row 3 col 1 in Fig. 7) can be located precisely, which leads to a well-aligned spatial evolution for videos before being compared.

In summary, the visualization tellingly depicts the capability of TA²N in correcting the misalignment.

Firstly, we break down our proposed TA²N into its component parts and compare the performance gain of the TTM and ACM when applied separately. Quantitative results on UCF101 and SSv2 datasets are listed in Tab. 3. We can observe that both TTM and ACM can boost the performance of few-shot action recognition with each stage playing an equally important role in action alignment. When TTM and ACM are applied in a two-stage manner, the performance is further improved, thereby supporting our notion of a two-stage sequential design.

Further, we split the spatial and temporal coordination parts in ACM. A single TC improves the baseline with 4.5 and 4.7 in UCF and SSv2 respectively. When TTM and TC are applied sequentially, the performance grows with a large margin. This proves that our proposed TTM and TC could well address the temporal misalignment from two distinctive aspects. Combined with SC, we can obtain the best performance, which advocates the necessary of aligning the evolution in spatial dimension. In a word, the above results point towards the fact that our TA²N provides an effective solution to address these two critical misalignment problems (as handled in the two stages) in few-shot action recognition task.

To reveal the effectiveness of spatial coordination (SC) module design, we compare our implementation (Mask-based) with other alternative designs. (1) Enumerate: it enumerates through all possible integer offsets in $\mathbb{Z}^{2}$, and straightly index the intersection area in feature $x,y$ coordinates. The offset with the minimum metric distance between support and query is considered the optimal one. It can be regarded as a simple baseline. (2) Grid-based: it generates grids according to our predicted offset and then uses the re-sampling trick to sample features in the intersection area. Their results are shown in Tab. 4. As we can see, our implementation outperforms the simple enumeration by a great margin, which also demonstrates the ability of our offset predictor in SC. Compared to Grid-based manner, our design is more straightforward and computationally tractable, with slightly better performance.

To further illustrate the effect of our proposed TA²N more intuitively, we visualize the feature embedding of query and the prototype of support before and after applying our framework by t-SNE method in Fig. 8. We can see that each cluster appears more concentrated and closer to its prototype after alignment. It further proves that our TA²N can well align the query videos to support videos and obtain a more consistent feature representation.

The improvement on the SSv2 dataset for each category using the proposed TA²N is presented in Fig. 9. What stands out in this figure is that the performance increases with a large margin for all categories. Moreover, some categories’ accuracy (e.g. “pouring sth out of sth”, “approaching sth”, “poking a stack of sth”) rose sharply ($>20\%$ improvement) using TA²N. Interestingly, these action classes are the ones that are more vulnerable to the misalignment problem. It also prove the advantages of temporal alignment in actions with limited information.

We describe the detailed structure of proposed offset predictor in Tab. 5.

Given predicted offset $o=[o_{x},o_{y}]\in\mathbb{R}^{1\times 2}$, we generate masks $m_{x}$ and $m_{y}$ for $x$ and $y$ coordinates respectively. Then we can obtain the 2D offset mask $I_{o}=m_{x}\times m_{y}$. Specifically, $m_{x}$ is calculated by this piece-wise linear function:

where $o_{x}$ is the offset in the $x$ dimension, the slope $\gamma=3$ in our case. $m_{y}$ is obtained in $y$ coordinate in the same way. The value of $\gamma$ ensures the width of the margin is about 1 pixel, which trades off between accurate result and differentiability. $m_{y}$ is conducted in $y$ coordinate in the same way.

The perturbations added on the offset are 8-directional vectors, with amplitude decays for every 40 epochs. We average the masks generated with different perturbations to get the final one. This perturbation is disabled in testing.