Improving Action Quality Assessment Using Weighted Aggregation

Let, $V=\{F_{t}\}_{t=1}^{L}$ be the input video containing $L$ frames, where $F_{t}$ denotes the $t^{th}$ frame. It is divided into $N$ non-overlapping clips, each of size $n=\lceil\frac{L}{N}\rceil$. Thus we define the $i^{th}$ clip as $C_{i}=\{F_{j}\}_{j=i\times n}^{i\times(n+1)-1}$. The feature extractor takes in a clip $C_{i}$ and outputs a feature vector $f_{i}$. For the feature extractor, we experiment with 3D ResNets[6] and (2+1)D ResNets[21] with varying depth and input clip size. Next, we aggregate these clip level features to obtain a global video level representation. Finally, a linear-regressor is trained to predict the score from the video level feature representation. Following the majority of previous works [16, 14, 18, 15], we model the problem as linear regression. This makes sense as the action quality score is a real number. To experiment with the relation of the ResNet feature extractor’s depth with the AQA pipeline’s ability to learn, we experiment with 3 different depths:

• •

  34 layer: We experiment with both 34 layer 3D and (2+1)D ResNets. The only difference being 3D ResNets use $3\times 3\times 3$ convolution kernels, on the other hand (2+1)D ResNets use a $1\times 3\times 3$ convolution followed by $3\times 1\times 1$ convolution[21]. We take the final average-pool layer output (512 dimensional) and pass it through 2 fully-connected layers having $256$ and $128$ units. The 3D ResNet takes input 16 frame clips. On the other hand, we experiment with 3 different variations of (2+1)D 34-layer ResNet, processing 8, 16, and 32 frame clips using available pre-trained weights,

• •

  50 layer: We experiment with 50 layer 3D and (2+1)D ResNets. The final average-pool layer outputs a feature vector of size $2048$. We take this feature vector and input it into 3 fully-connected layers having $512$, $256$, and $128$ units. Only 16 frame clips are processed.

• •

  101 layer: We experiment with 101 layer 3D ResNet. The remaining details about the input clip size and output feature vector processing are identical to the 50 layer ResNets.

Most of the previous works dealing with AQA process the entire input video by first dividing it into multiple smaller clips of equal size, due to memory and computational budget. Most CNNs are designed to process 8, 16, or 32 frames at once. Then the features extracted by the CNN are aggregated to form a video level feature description. Next, a linear-regressor predicts the final score based on this feature description.

The best performing works aggregated the clips by simply averaging them [16, 19, 15]. Some other works [15, 14] aggregated using LSTMs[7]. However, LSTM networks, although make sense in theory because of their ability to handle time sequences, perform worse due to the lack of big-enough datasets dedicated to AQA.

We propose that simply averaging the clip-wise features is an ineffective measure. It should not be able to preserve the temporal information available in the data. This follows from the fact changing the order of the clip level features will generate the same average and hence the same score prediction. Furthermore, expert human judges focus more on mistakes and deviations of the performers and these have a bigger impact on the score. Hence we think, a weighted averaging technique might be more suitable, as the linear-regressor will be able to base its decision on features more important from each clip.

More concretely, if the feature vector extracted from clip $C_{i}$ is $f_{i}$, we propose the video level feature vector as

where $w_{i}$ is a weight vector corresponding to the feature vector $f_{i}$ and $\odot$ represents Hadamard Product or elementwise multiplication. $w_{i}$ is of the same dimensions as $f_{i}$ and learned using a small neural network of 4 layers. This smaller neural network takes as input 128-dimensional feature vector $f_{i}$ and runs it through fully connected layers containing 64, 32, 64, and 128 neurons. All but the final layers employ a ReLU activation function. The architecture is explained in Figure 2. Finally, to ensure the weights corresponding to the same element of different weight vectors sum up to one, a softmax is applied along with the corresponding elements of all the weight vectors. We call this shallow neural network Weight-Decider(WD).

Finally, the linear-regressor can predict the score using the feature vector $f_{video}$ as proposed by equation 1.

The proposed WD module can be used with any of the feature extractors we planned to use in our experiments. WD replaces averaging as aggregation within the typical AQA pipeline. It can be trained with the rest of the AQA model in an end-to-end manner.

Adding WD to the AQA pipeline to replace averaging as aggregation does not cost much in terms of computational resources. To see this, recall that the WD module has 3 hidden layers and an output layer, consisting of 64, 32, 64, and 128 neurons in that order. Each of these layers contains a weight matrix and a bias. The first hidden layer takes 128 inputs and has 64 neurons. Thus the weight matrix is of dimensions $64\times 128$. Hence, number of trainable parameters in this layer (including the bias term) is $(64\times 128+1)=8193$. Similarly, the second hidden layer, third hidden layer, and the output layers contain $2049$, $2049$, and $8193$ trainable parameters correspondingly. By summing up all the trainable parameters in each layer, we can see that WD only contains $20484$ trainable parameters. Spatio-temporal ResNet feature extractors have millions of trainable parameters. Hence, additional resources required to train the WD module used in an AQA pipeline are not much. As an example, the 3D and (2+1)D ResNet-34 feature extractors we are using contain 63.6 million trainable parameters [12]. Thus, using the WD module on top of ResNet-34 would increase the number of trainable parameters by approximately $0.03\%$. The ratio would be smaller for a deeper ResNet. Hence, our proposed WD does not require many computational resources to be incorporated into the pipeline. Our experiments support this. We found the training time for 32-frame ResNet(2+1)D-34 to be 4557 seconds per epoch, and the inference time to be 604 seconds per epoch. The introduction of WD increases the training time per epoch by 24 seconds (0.52%) and the inference time by 21 seconds (3.47%).

The biggest dataset dedicated to AQA. It contains 1412 video samples split into 1059 training and 353 test samples. The samples are collected from 16 Olympic dive events. Each sample is 103 frames long and accompanied by the final action quality scores from expert Olympic judges. It was published by Parmer and Morris[16]. We used the exact same train/test split made public in their work[16].

In line with previously published literature, we use Spearman’s rank correlation as the evaluation metric. This metric measures the correlation between two sequences containing ordinal or numerical data. It is calculated using the equation:

$\rho$ = Spearman’s rank correlation
$d_{i}$= The difference between the ranks of corresponding variables
$n$ = Number of observations

We implemented our proposed methods using PyTorch[17]. All the 3D ResNets and (2+1)D ResNets processing 16 frame clips were pre-trained on Kinetics-700[9] dataset¹¹1Weights available at: https://github.com/kenshohara/3D-ResNets-PyTorch. The (2+1)D Resnets processing 8 and 32 frame clips were pre-trained on IG-65M dataset[4] and fine tuned on Kinetics-400[9] dataset²²2Weights available at: https://github.com/moabitcoin/ig65m-pytorch.

For each ResNet, we separately experimented using both averaging and WD as feature aggregation. We temporally augmented by randomly picking an ending frame from the last $6$ and chose the preceding $96$ frames for processing. The frames were resized to $171\times 128$ and center cropped to $112\times 112$. Random horizontal flipping was applied. Batch-normalization was used for regularization. We defined the loss function as a sum of L2 and L1 loss between the predicted score and ground-truth score as Parmar and Morris[16] suggested. We trained the network using the ADAM optimizer[10] for $50$ epochs. We used a learning rate of 0.0001 for modules with randomly initialized weights and 0.00001 for modules with pretrained weights. Train and Test batch sizes were 2 and 5.

In Table 1(a), we present the experiment results of varying the depth of the ResNet feature extractor and the aggregation scheme. We can see that 34 layer (2+1)D ResNet with WD as aggregation performs the best with a Spearman’s correlation of $0.8990$. Increasing the depth to 50 layers somewhat decreases Spearman’s correlation. However, the results are still competitive. At 101 layer depth, even when initialized with pretrained weights from Kinetics[9], overfitting occurs fairly quickly. The overfitting is also evident from the train/test curves presented in Figure 4. The likely reason behind this is the increased number of parameters in the feature extractor. This leads us to establish that the current biggest AQA dataset has enough data to train a 34-layer and 50-layer ResNet feature extractor with generalization, however it overfits the 101-layer ResNet feature extractor.

Because of how (2+1)D ResNets are designed, they have a similar parameter count to their 3D counterparts [21]. Because the overfitting is occurring due to the high parameter count, we do not repeat the experiment with a (2+1)D ResNet-101 feature extractor.