FCA-RAC: First Cycle Annotated Repetitive Action Counting

As shown in Figure.2, assuming the input video contains $L$ frames $V=[v_{1},v_{2},...v_{L}]$, the first $N$ frames $V_{1}=[v_{1},v_{2},...v_{N}]$ are labeled as the first action cycle. We sample these N frames to k frames (k = 4 in this article) to represent the action’s start, middle, and end stages. The sampling rate can be calculated as $R=k/N$. Consequently, the sampled video ( $V_{s}=[v^{\prime}_{1},v^{\prime}_{2},...v^{\prime}_{F}]$, $F=R*L$) contain ${F}$ frames. By sampling all videos according to the speed of the first action, we ensure consistency in action speeds across different videos, thereby mitigating the effect of speed variations.

We extract the video sequence $V_{s}$ with sliding window step size $\frac{k}{2}=2$, as in Eq.1.

For feature extraction, we employ the Video-Swin-Transformer[26], which is designed specifically for visual recognition tasks.

Let the video vector $V^{\prime}$ pass through the Video-Swin-Transformer. For each clip, the size of the extracted features is $7\times 7\times\frac{k}{2}\times 768$, which can then be concatenated to form a tensor of size $7\times 7\times F\times 768$.

We then apply a conv3D layer with $3\times 3\times 512$ filters and ReLU activation, to enhance the combination of spatial and temporal features. A spatial pooling layer aggregates space information into the channel dimension, resulting in the final temporal context features X as defined below:

To capture the similarities between the first action cycle and the entire video, we introduce the multi-temporal granularity convolution (MTGC) module. The MTGC module leverages the spatio-temporal information contained in the first action cycle to estimate the count for the entire video. We use the first action cycle $X^{\prime}=[x_{1},x_{2},x_{3},x_{4}]$, as a temporal granularity convolution kernel, to convolve with the entire video feature $X$. To account for the variations in action speed within the video, we interpolate the first action cycle kernel to 3 and 5 in different scales, respectively, resulting in three multi-scaled kernels. Each kernel performs temporal granularity convolution on the entire video with stride 1, and their outputs are concatenated to obtain the temporal granularity feature $G\in\mathbb{R}^{F\times 3}$.

In the pre-training stage, $G$ is passed through the prediction layer, which consists of three 1D convolution layers and the Tanh activation function. The output is the final density map $D=[d_{1},d_{2},...d_{F}]$ representing the action period distribution. By summing up the density, we obtain the action count in the video.

Density Map The advantage of the density map is the strong interpretability [27] [28] [29] [30], with each frame representing the action’s distribution. To construct the density map of the first action cycle, we follow the procedure in [1], using the Gaussian function [31] with a 99% confidence interval. Since a 99% confidence interval corresponds to $\mu\pm 3\sigma$, we could obtain $G_{\mu,\sigma}(x)$ from $[1,k]$. Then the density value $d_{i}$ can be calculated as follows:

This approach allows us to construct an accurate density map of the first action cycle to train the model.

In the proposed Training Knowledge Augmentation (TKA) strategy, we leverage the similarities of the first action cycle between the input video and training set to enhance model performance. After pre-training, we construct an embedding vector space $e\in\mathbb{R}^{T\times k\times D}$ capturing the embedding vectors from the first annotated cycle of each video in the training set, where T is the total number of the training videos, and k is the number of sampled frames of the first cycle (i.e. $k=4$ in this article), and D stands for the dimension of the embedding vectors. Each embedding vector $e_{i}\in\mathbb{R}^{k\times D},i\in 1,2,...T$ is obtained by passing the sampled first action of the training video through the Encoder described in Sec.3.2. During both fine-tuning and inference stages, the top K closest embedding vectors are selected based on their similarity to the temporal context features of the first cycle $X^{\prime}$ from the input video. The Euclid distance between $X^{\prime}$ and $e_{i}$ can be defined as follows:

Then we sort the distances d in ascending order and select the top $K$ vector with the smallest distance. Therefore, we can express the process of selecting the $K$ closest vector using the formula:

Here $e_{(i)}$ represents the $i$th closest vector to $X^{\prime}$ after sorting all values in $d$. Subsequently, each embedding vector $e_{(i)}$ performs MTGC with the entire video, as described in Sec.3.3. After concatenating the output of the TKA module with the original output of MTGC, we obtain the feature $G^{\prime}\in\mathbb{R}^{F\times 3\times(K+1)}$.

To compress the resulting feature, we employ a feature fusion strategy using an attention-pooling layer[32][33] with learnable weights:

Finally, the feature of $G^{\prime}\in\mathbb{R}^{F\times 3}$ is passed through the prediction Layer to generate the density map.

To incorporate the label of the first cycle, we crop the first action from the output density map $D\in\mathbb{R}^{F}$, which is denoted as $D^{1st}$. We use the Gaussian function $G(x)$ to derive the ground truth density of the first cycle $Gt^{1st}$. Therefore, we can calculate the MSE loss (Mean Squared Error) of the first cycle:

Additionally, we compute the MAE loss (Mean Absolute Error) between the ground truth count $G^{count}$ and the predicted count $D^{count}=\sum(D)$ obtained by summing the density map values:

The overall loss function is a linear combination of the MSE and MAE losses, weighted by $\alpha$ :

Given a video clip, since its first action cycle is labeled, we first employed our sampling technique to sample the video. Then the sampled video is fed into the model (with TKA strategy), shown in Figure.2. To apply the TKA strategy, the top K closest embedding vectors from the training set are selected, and each embedding vector serves as a convolution kernel to perform MTGC with the input video. After concatenating, the resulting features are passed through the attention-pooling and prediction layers to generate the density map. After applying a linear sum of the density map, we could obtain the predicted value of the action period.

The setting of our framework is different from the previous one [1][12]. These methods only use the annotation from the training set, and predict the action number in the test set without annotation, while our framework utilizes the annotated first action in the video during prediction. For a fair comparison, we introduce two baseline modules for action counting prediction. Each baseline utilizes temporal context feature $X$ as input and generates a video density map by passing the features through different module types.

The first baseline, named First Cycle Video Attention (FC-V), employs an attention mechanism based on the first annotated cycle. Given the temporal context features $X$, we use the first action cycle as the query matrix Q, while the subsequent temporal features are treated as the key matrix K and value matrix V. These matrices are processed through trained linear layers. The cross-attention feature C is calculated by applying the attention equation:

The cross-attention feature $C$ is then fed into the prediction module to generate the output density map.

The second baseline, named Video to Video Attention (V-V), employs a self-attention mechanism to compute the similarity matrix from the temporal context features $X$, similar to the approach used in TransRAC [1]. V-V is designed to leverage the entire temporal context of the video, which may provide useful information about the frequency and duration of the repetitive actions. Specifically, the feature $X$ is multiplied with two weights matrices to obtain keys matrix K and query matrix Q. Then the similarity matrix $S$ is calculated by $S=f(Q,K)$ where $f$ denotes the dot product.

Unlike TransRAC[1], we apply a $3\times 3$ convolution layer to the similarity matrix $S$, yielding the context feature $[32\times F\times F]$. Afterward, an adaptive average pooling layer modifies the last axis from $F$ (the frame length) to 16, resulting in the context feature C. Finally, the context feature $C$ is fed into the prediction module to generate the output density map.

Test Time Adaptation. To enhance the performance of both baselines, we incorporate the Test Time Adaptation using the approach from[2] . This adaptation strategy adjusts the network to the input video during test time, further improving the accuracy of the estimated count, as shown in the left of Figure.1b. Specifically, during test time, the model parameters are frozen except for the prediction module. We perform several gradient descent steps on the first action cycle of the testing video to optimize the predicted count with the mean-squared error loss defined in Eq.8.

We use 4 repetition counting datasets [12, 1, 11, 37] to evaluate the effectiveness of our framework. The summary of their statistics is shown in Table.2.

RepCount-A The RepCount-A dataset [1] consists of 1041 videos with about 20,000 fine-grained annotations total. It includes videos of varying lengths and multiple anomaly cases, with each video annotated for the beginning and end of each action period. The average video length in the dataset is 39.35 seconds, significantly surpassing the durations of UCFRep[12] and CountixAV[11]. On average, each video clip in the dataset contains about 16 action cycles.

Countix-AV[11], a subset of Countix[9], is a dataset sourced from YouTube. It consists of 1863 videos, with 987, 311, and 565 videos allocated for training, validation, and testing, respectively. Since each video only annotates the count of repeated actions, which does not meet our task requirements, we manually annotate the start and end frames of the first action cycle to align with our framework.

UCFRep[12], a subset of UCF101, is a dataset that selectively includes 23 action categories out of the original 101, focusing on cyclically performed actions. The dataset contains a total of 526 videos, with 421 videos assigned to the training set and 105 videos designated as the validation set. Each video in UCFRep is annotated with the temporal boundaries of each repetitive action, making it directly applicable to our task.

QUVA [37], comprises 100 videos exhibiting various activity ranges. Due to its relatively small scale, the QUVA dataset is commonly employed for evaluation purposes. All videos in the QUVA dataset are annotated with the temporal bounds of each interval, enabling us to assess the generalizability of our method.

To measure the accuracy of our method, we adopted two evaluation metrics following the previous work [9][12]: Mean Absolute Error (MAE) and Off-by-One accuracy (OBO). MAE represents the normalized absolute error between the predicted count and the ground truth count, while OBO measures the accuracy of the repetition count across the entire dataset. Specifically, for OBO, we considered a video to be counted correctly if the predicted count is within one count of the ground truth; otherwise, it was considered a counting error. The definitions of MAE and OBO are as follows:

where $G_{i}^{count}$ and $D_{i}^{count}$ represent the ground truth and predicted counts of the $i$-th video, respectively, and $N$ is the total number of videos in the evaluation set.

We implement our method using PyTorch with four NVIDIA V100 GPUs. The Video Swin Transformer tiny [26] served as the encoder, which was pre-trained on the Kinetics400. Due to the limitation of GPU memory, the parameter of the pre-trained encoder is frozen. To train our FCA-RAC model, we normalized the first annotated cycle to $k=4$ frames and resized all frames to $224\times 224$. Within a batch, we padded the sampled frames to match the longest selected frames for training purposes. The density map in the padded frames was set to 0, so the predicted number of actions would be the sum of the ”real” density map. We used a weighted combination of $L_{MSE}$ and $L_{MAE}$ for the loss function with weight coefficients $\alpha=10$. We train the model for 16K steps with a learning rate of $8\times 10^{-6}$ and optimized by the Adam optimizer.

After pre-training, we applied TKA to fine-tune the model. Before fine-tuning, we save the encoded feature of the first action cycle of all training videos to construct the embedding vectors. During each step of the fine-tuning, the top K closest vectors with the first action of the input video are selected to perform MTGC and the subsequent modules to generate the density map. The embedding vectors were updated at every epoch to align them with the output of the model. We perform 1.6K steps during fine-tuning with a learning rate of $2\times 10^{-6}$.

To evaluate the performance of our FCA-RAC model, we conducted comparisons with existing methods [34, 35, 26, 36, 12, 1] in Table.1. The results reported for RepCount-A and UCFRep datasets are taken from [1, 38]. Since the countixAV dataset lacks label information for action cycle boundaries, we manually annotated the start and end frames of the first action cycle.

Our FCA-RAC model demonstrates superior performance on both the RepCount-A and countixAV datasets. In RepCount-A, the model achieves an MAE of 0.268 and an OBO of 0.47, which outperforms the TransRAC[1] by 0.175/0.18 on MAE/OBO. Similarly, in countixAV, our model attains an MAE of 0.330 and an OBO of 0.58, slightly surpassing the result in [11]. Notably, the FCA-RAC model’s performance on the UCFRep dataset is also comparable with the state-of-the-art.

Our model’s experimental setting differs from previous methods. While previous methods lack label information in the test set, we utilize annotated first action cycles in testing videos to enhance generalizability. To ensure a fair comparison, we also evaluate the baseline described in Sec.3.8, as shown in Table.1. Our model exhibits MAE/OBO performance gains of 0.076/0.11(0.054/0.11), 0.062/0.02(0.049/0.03), 0.155/0.21(0.061/0.08) over FC-V(V-V) on RepCount-A, CountixAV, and UCFRep dataset, respectively. These results highlight our model’s significant superiority over the baseline across all three datasets. These findings provide strong evidence that our FCA-RAC is an effective model for repetition action counting.

Generalization We evaluated the generalization capabilities of our FCA-RAC model from two perspectives.

First, we conducted experiments on the UCFRep and QUVA datasets using the model pre-trained on the RepCount-A dataset. The results are shown in Table.3. Our methods outperform the TransRAC[1] by 0.339/0.23, and 0.512/0.51 in terms of MAE/OBO on UCFRep and QUVA datasets respectively. These results demonstrate the superior generalization capabilities of our method compared to previous approaches when applied to unseen datasets.

Second, we reorganized the RepCount-A dataset according to the guidelines in [1], creating new training, validation, and test subsets with disjoint action types. This ensured that the actions in the test set did not appear in the training set. We then evaluated our FCA-RAC model on this newly constructed dataset, and the results are presented in Table.4. Our method achieves improvements of 0.175/0.18, and 0.210/0.13 in terms of MAE/OBO in the regular setting and resplit setting respectively, which demonstrates that our model also exhibits strong generalization performance on previously unseen actions. These findings demonstrate the strong generalization performance of our model when faced with diverse and unfamiliar action scenarios.

In this section, we conduct some ablation experiments to evaluate the effectiveness of the designing of the FCA-RAC model. We train the model on the training set of RepCount-A, countixAV, and UCFRep datasets, and then evaluate their performance on the corresponding test set.

Multi-Temporal Granularity Convolution. We investigated the impact of applying temporal granularity convolution to our model to accommodate speed variations within a video. Table.5 demonstrates that MTGC with a kernel scale ranging from 3-5 achieves the best results. By using the multi-scale of the first action cycle as convolution kernels, we improve action counting performance by capturing action variations across different time scales within a video. However, a kernel scale ranging from 2-6 resulted in inferior performance, suggesting that kernels with extreme sizes may not effectively capture the action patterns in the video and disrupt the predicted results. Moreover, compared with the result of the FC-V baseline, it can be seen that the use of MTGC significantly improves the performance of the action counting task.

Training Knowledge Augmentation. Furthermore, we explored the effectiveness of Training Knowledge Augmentation (TKA). Figure.3 demonstrates that fine-tuning and inference with TKA surpass the performance without TKA. Moreover, we observed notable performance improvements by increasing the number of action cycles from 1 to 10 during TKA. Table.4 reveals the performance of TKA and non-TKA scenarios under different data settings. In the regular setting of the RepCount-A dataset, the TKA strategy gains 0.048/0.09 on MAE/OBO. While in the re-split setting, the corresponding values were 0.108/0.08. The results illustrate the strong generalization ability of TKA when dealing with both seen and unseen actions, emphasizing the importance of incorporating multiple cycles in TKA to enhance overall model performance.

Feature Fusion We analyzed the performance variations resulting from different feature fusion strategies on TKA in Table.6. The max pooling method, which selects the closest feature for TKA, is identical to TKA (top1) shown in Figure.3. The result reveals that feature fusion with attention pooling achieves the best performance. This observation indicates that learnable fractions from diverse features within TKA play a role in enhancing the model’s performance.

Loss Term As we use two losses to train the model, we compare the performance with different values of $\alpha$. As shown in Table.7, while training the model exclusively with Mean Absolute Error (MAE) yields effective results, incorporating Mean Square Error (MSE) further enhances performance. Additionally, through experimentation, we determined that the optimal value for $\alpha$ is 10.

In this part, we give some qualitative results to better characterize the ability of our framework.

In Figure.4, we show some nearest actions selected using our proposed TKA strategy, given an input action. For seen actions (Figure.4a), the nearest neighbors belong to the same action type, i.e. front raise and push up. For unseen actions (Figure.4b), the nearest neighbors are selected from the samples with similar actions. For example, given an input action of jumping jack, the selected actions are raise up and squat. Similarly, given an input actions rolling scope, the selected action is front raises. This visualization indicates that the TKA strategy successfully selects the most useful action in the training set to enhance the training and inference performance.

In Figure.5a, we present successful cases achieved by our method. In the first instance, the subject executes a sit-up with relatively stable frequency, allowing our model to precisely predict the density. In the second example, the individual performs a squat with gradually decreasing speed due to fatigue; nonetheless, our model can also accurately predict the density map. While our framework performs well on the majority of data, there are still instances of failure, as depicted in Figure.5b. In the first pair, a girl performs jumping jacks, and the video speed abruptly slows down in the latter half, approximately one-third of the speed compared to the first half. This big change in video speed disrupts our model’s prediction. In the second pair, a boy plays tennis on the ground, and the camera movement is rapid, capturing the boy running and hitting the tennis ball from various angles. These extreme cases can adversely affect our model’s performance.