FACTS: Fine-Grained Action Classification for Tactical Sports

The development of AI referees for fencing has largely focused on basic action classification, such as determining touches for the left or right fencer or identifying simultaneous actions. One of the earliest efforts in this domain was by Sholto Douglas [3], who utilized a fully recurrent convolutional network combined with a pre-trained InceptionV3 model. This approach incorporated optical flow to capture the relative movement of fencers, achieving an accuracy of approximately 60%. Douglas also developed an efficient method to automatically scrape and label fencing videos using the digital scoreboard, which set the stage for subsequent research.

Building on this foundation, researchers explored pose estimation to refine action classification. For example, GitHub user GalDude33 [5] improved accuracy to 70% by leveraging pose estimation as a natural filter, isolating the fencers’ movements from the background. Similarly, Alexandre Pageaud [6] adopted a CNN-LSTM architecture that also utilized pose estimation, achieving comparable results.

Recent advances have incorporated temporal convolutional networks (TCNs) and multimodal inputs. The Allez Go model [7] combined TCNs with audio data to enhance performance, achieving an accuracy of 90%. This integration of audio allowed the model to detect moments of blade contact, addressing the challenges posed by the thin, fast-moving fencing blades that are difficult to capture visually. Another innovative approach by Sunal et al. [8] used YOLOv3 and ResNet-34 to classify specific actions like counter-attacks and lunges, using only still images as input. This departure from video-based methods questioned the necessity of temporal information for action classification.

Pose estimation has also been used alongside rule-based systems to classify fencing actions. Zhou and Smolin [9] developed a manually tuned decision-making algorithm that extracted features like velocity and acceleration from pose data. While effective in structured scenarios, such methods struggle with edge cases and lack generalizability. Despite progress in AI refereeing for fencing, reliance on pose estimation and the absence of robust datasets have limited the field’s advancement.

Boxing-related research has predominantly focused on punch classification and action recognition, leveraging pose estimation, depth imagery, and skeleton-based methods. Bhargav [10] developed a skeleton-based action recognition model using PoseConv3D, fine-tuned on a custom dataset with six punch types, achieving an accuracy of 87.32%. Their approach relied on extracting skeletal key points through Faster-RCNN and HRNET models, which enabled precise action classification for shadow boxing and heavy bag training. Similarly, Kasiri et al. [11] presented a robust framework utilizing overhead depth imagery and hierarchical SVM classifiers for fine-grained punch recognition, achieving 97.3% accuracy on a dataset of elite boxers. Their method incorporated body part detection, trajectory features, and fusion techniques to distinguish subtle variations in punch types. These studies highlight the effectiveness of combining pose data and depth features, although reliance on specialized equipment and occlusion challenges remain limitations. Recent advancements emphasize the potential of integrating deep learning and multimodal data for improving boxing analytics and enhancing performance feedback systems.

Transformer models have transformed action recognition by leveraging self-attention mechanisms to model long-range dependencies and spatiotemporal interactions in video sequences, offering advantages over traditional CNNs and RNNs. Key advancements include Yan et al.’s Multiview Transformers [12] for spatial-temporal modeling, Bertasius et al.’s space-time attention framework [13], and Liu et al.’s Video Swin Transformer [14], which improved efficiency with hierarchical window mechanisms.

Hybrid models like Kalfaoglu et al.’s combination of 3D CNNs with BERT [15] and Arnab et al.’s ViViT model [16] further extend transformer capabilities. Multimodal approaches such as TREAR’s RGB-D integration [17] and Das et al.’s pose-driven attention [18] highlight their adaptability to diverse inputs. In skeleton-based recognition, models like Zhang et al.’s Spatial-Temporal Specialized Transformer [19] effectively capture joint-level dynamics.

Challenges remain, including computational costs and reliance on large datasets, addressed by solutions like sparse attention mechanisms [20]. Overall, transformers provide robust tools for fine-grained action recognition, excelling in long-sequence processing and multimodal integration.

In this study, we introduce a novel and meticulously curated fencing dataset focused on video action classification. The fencing dataset, sourced from Quarte Riposte, comprises official competition footage videos across 16 unique action classes. Unlike existing datasets, which are often geared toward pose estimation or full-match analysis, our dataset offers precise, pre-clipped, and labeled fencing action videos, making it uniquely suited for advanced video action research.

The 13,459 video clips were annotated through a majority vote by members of the fencing community, capturing the nuances and representing a broad consensus on technique. To ensure the highest quality, conflicting or sparse annotations were systematically excluded to maintain label accuracy and classes with insufficient samples were removed to enhance dataset reliability.

Our dataset preparation also included data augmentation through horizontal flipping, effectively doubling the dataset size while preserving the integrity of the action semantics. The final dataset contains 6,400 clips compared to the original 13,495 clips and features eight well-defined labels:

• •

  Attack Left (AL): offensive move directed from the left fencer.

• •

  Attack Right (AR): offensive move directed from the right fencer.

• •

  Riposte Left (RL): counter after a parry from the left fencer.

• •

  Riposte Right (RR): counter after a parry from the right fencer.

• •

  Counter-attack Left (CAL): intercepting attack to disrupt the opponent’s offense from the left fencer.

• •

  Counter-attack Right (CAR): intercepting attack to disrupt the opponent’s offense from the right fencer.

• •

  Remise Left (ReL): immediate follow-up thrust without withdrawing from the left fencer.

• •

  Remise Right (ReR): immediate follow-up thrust without withdrawing from the right fencer.

This dataset is the first comprehensive resource tailored to fine-grained fencing action analysis. Compared to prior works that emphasize pose data [6], footwork [21], or isolated images [22], this dataset provides a full spectrum of clipped and labeled actions. Furthermore, our work expands the number of action classes from the standard five [8] to include nuanced distinctions such as left/right attacks, ripostes, and remises and prioritizes clean, consensus-driven annotations. The structured design facilitates advanced deep-learning approaches and encourages further exploration in fine-grained action recognition.

The boxing dataset, sourced from the Olympic Boxing Punch Classification Video Dataset [23, 24, 25, 26, 27], includes videos across 8 distinct action classes. Captured during a 2021 boxing league, the footage was recorded using four GoPro cameras in Full HD resolution at 50 frames per second. Each camera setup included a 128 GB micro SD card, powerbank, and tripod to ensure uninterrupted, high-quality recordings from multiple angles.

Each action was then annotated frame by frame by licensed boxing referees, with precise labels and coordinates assigned to classify each punch type. In this study, we cut up all the clips for each action for a final collection of 8,000 action clips prepared for our model. The final 8 labels with descriptions are:

• •

  Left Hand Head Punch (LHHP): direct punch to the opponent’s head using the left hand.

• •

  Right Hand Head Punch (RHHP): direct punch to the opponent’s head using the right hand.

• •

  Left Hand Missed Punch (LHMP): missed punch attempt with the left hand.

• •

  Right Hand Missed Punch (RHMP): missed punch attempt with the right hand.

• •

  Left Hand Block Punch (LHBlP): defensive block using the left hand.

• •

  Right Hand Block Punch (RHBlP): defensive block using the right hand.

• •

  Left Hand Body Punch (LHBP): punch directed at the opponent’s body with the left hand.

• •

  Right Hand Body Punch (RHBP): punch directed at the opponent’s body with the right hand.

This dataset enhances the original Olympic Boxing Punch Classification Video Dataset by segmenting it into clear, labeled clips for each action class, optimizing it for machine learning models. Unlike other datasets that focus on foul classification or rely on pose estimation, this dataset targets precise punch actions without requiring pose data, reducing potential inaccuracies. It is uniquely structured to support fine-grained action recognition, making it a valuable resource for advancing close-combat sports analysis.

Overall, the fencing and boxing datasets offer unique contributions to sports video analysis, specifically for fine-grained fast-paced close-combat sports. Fully clipped and labeled, this data is suitable for advanced machine-learning applications.

Our model employs a transformer-based architecture pre-trained with Video Masked Autoencoders (VideoMAE)  [28, 29], an adaptation of the Masked Autoencoders (MAE)  [30] for video data. VideoMAE captures both the spatial and temporal information in a scene by reconstructing missing patches across frames. Although already shown to work well for video classification[31, 32, 33, 34, 35, 36], this approach proved critical for learning fine-grained actions in dynamic sports like fencing and boxing.

The model architecture is based on the Vision Transformer (ViT) [37], with enhancements to process temporal dimensions. The encoder learns latent representations from masked video frames, while the decoder reconstructs the pixel values of these masked patches, improving the model’s ability to interpret complex motion patterns. This capability is critical for learning subtle differences in fast-paced, close-combat sports actions. This approach is adopted without relying on complicated skeleton extraction, pose estimation, or external body-worn markers or sensors.

From Table I, we can see that across the different fencing labels, the frame counts are relatively consistent, however, subsampling is still needed to ensure uniform input lengths. Opposite fencing, the boxing dataset was 15 frames across the board for all labels and all videos.

Uniform temporal subsampling was performed as described in (1), ensuring uniform input lengths for the transformer neural network:

where $T$ represents the total frame count, and $X_{\text{sub}}$ is the subsampled set of frames.

Spatial preprocessing was tailored to each dataset. Fencing videos, due to the horizontal nature of the sport, were padded on the top and bottom to match the horizontal dimension before resizing to a target resolution of 640x640 pixels. Boxing videos were uniformly padded and resized to ensure consistency across samples. Padding was applied as shown in (2):

Standardization, defined in (3), was applied to normalize pixel values to zero mean and unit variance across channels, enhancing training stability:

where $\mu$ and $\sigma$ represent the mean and standard deviation of pixel values.

By directly processing raw video data, our model circumvents the inaccuracies and inefficiencies associated with pose estimation, enabling robust classification of rapid and occluded movements.

To fine-tune VideoMAE, we implemented a carefully designed training pipeline optimized for video-based data. Both datasets were stratified into training, validation, and testing subsets, ensuring balanced class distributions to improve generalization across action categories.

Preprocessed videos were normalized using ImageNet mean and standard deviation, subsampled to 16 frames per clip and resized to the model’s expected input dimensions of 224x224 pixels. Training utilized a batch size of 4, a learning rate of $5\times 10^{-5}$, and gradient accumulation steps of 2, effectively simulating a batch size of 8 to accommodate GPU memory constraints. A warm-up ratio of 0.1 was applied during the initial training phase to stabilize optimization, with the model trained for 10 epochs and evaluated every 500 steps to select the best-performing checkpoint. Training was conducted on two NVIDIA GeForce GTX 1080 Ti GPUs with approximately 20 GB of combined VRAM.

The model architecture uses the MCG-NJU/videomae-base checkpoint pretrained on the Kinetics-400 dataset. This transformer-based architecture includes 12 layers, 12 attention heads, a hidden size of 768, and a feedforward network size of 3072, enabling it to effectively capture spatiotemporal dynamics in video data.

We prioritize reproducibility by explicitly documenting all training parameters, including batch size, learning rate, gradient accumulation steps, warm-up ratio, and evaluation intervals. Our preprocessed datasets are publicly available, providing a straightforward pipeline for replicating our methodology and extending it to other fine-grained action recognition tasks.

Our model’s performance was evaluated on separate datasets for fencing and boxing, with distinct classification tasks tailored to each sport.

As this paper introduces the new fencing action dataset that is being evaluated, there are currently no existing benchmarks for direct comparison. So, to provide a baseline, we also implemented a standard pose estimation model as described by Pageaud and GalDude33 [5, 6]. Table VI summarizes the classification accuracy for both the pose estimation model and our transformer-based model with pertaining.

The transformer model’s accuracy of 90% for fencing far exceeds the 64.8% achieved by the pose estimation model, underscoring the transformative potential of using transformer-based architectures for fine-grained action recognition.