A New Perspective for Shuttlecock Hitting Event Detection

TrackNetV2 Sun et al. (2020) is an advanced computer vision model designed specifically for object tracking tasks. Building upon its predecessor, TrackNetV1, this updated version offers enhanced performance and improved accuracy in tracking objects across video sequences. It employs deep learning techniques, particularly convolutional neural networks (CNNs), to extract relevant features from consecutive frames of a video. These features are then used to predict the location and trajectory of a specified object in subsequent frames. By learning the spatiotemporal patterns and motion characteristics of the tracked object, TrackNetV2 can accurately follow its movements over time.

After obtaining the predicted badminton trajectories from TrackNetV2, further processing steps are performed, including removing extreme offset points, curve fitting, and interpolation. Once the data is prepared accordingly, roughly rally segmentation and event detection are applied to identify specific events within the processed videos.

Court detection in Coach AI plays a crucial role in enhancing the accuracy and effectiveness of the system, enabling it to provide valuable insights and actionable information based on the specific dynamics and requirements of the sports court.

MoveNet is a deep learning model designed for human pose estimation and tracking. It is specifically developed to accurately detect and track human body movements in real-time from video or camera input. This will be further used to analyze the BallType played by the players.

OpenPose is a computer vision library and framework for real-time multi-person keypoint detection and pose estimation. It enables the accurate estimation of human body poses, including the positions of body joints such as the head, shoulders, elbows, wrists, hips, knees, and ankles. Using OpenPose nodes in this competition allows for more precise identification of the location of the hitter and the defender.

Video processing is a critical step to improve the score in the entire task. In this process, we first calculate the optical flow images using the method of Optical Flow Calculation embedded in Reynolds Transport Theorem¹¹1A Prior Feature Enhanced Layer for Small Objects Detection and Tracking Bruhn et al. (2005); Harouna and Mémin (2017) to capture the prior features of the video, then, we remove the background information to achieve a mechanism similar to attention as shown in Figure 1. And we will use the term ”optical flow video without background” to specifically describe this type of video. Subsequently, we input these processed video into SwingNet for shuttlecock hitting event detection.

SwingNet McNally et al. (2019) plays a crucial role in the field of computer vision and sports analytics, specifically in the domain of golf. Its application enables more efficient and detailed analysis of golf swings, contributing to the overall development and understanding of the sport. In here, we utilize the SwingNet, a deep learning model combining with the MobileNetV2 and bidirectional LSTM structure, for shuttlecock hitting event detection. That is, utilizing SwingNet enables us to extract the ShotSeq and HitFrame features for the desired csv file. Figure 2 demonstrates the inferred event probabilities for a video clip.

ViT, short for Vision Transformer Dosovitskiy et al. (2020), is a deep learning model that applies the transformer architecture to computer vision tasks. In contrast to the CNNs’ approach, ViT adopts a distinct strategy by leveraging the transformer architecture, initially designed for tasks in natural language processing. Moreover, the strength of ViT lies in its attention mechanism, as depicted in Figure 3, accompanied by the corresponding attention map illustrated in Figure 4. Here, we utilize ViT-B/16 to extract the information including Hitter, RoundHead, Backhand, BallHeight, BallType, and Winner from the videos.

The YOLO-series detector is a renowned family of object detection models. Specifically, YOLOv5 Zhu et al. (2021) is a popular and highly efficient object detection model, which bilds upon the success of its predecessors, YOLOv1, YOLOv2, and YOLOv3, and introduces several improvements to achieve better performance in terms of accuracy and speed. We utilize YOLOv5m to extract the information for LandingX, LandingY, HitterLocationX, HitterLocationY, DefenderLocationX, and DefenderLocationY.

To begin with, we opted for YOLOv5m as our chosen detection model. The rationale behind not selecting newer models like YOLOv7 is based on my experience. While YOLOv7 may achieve higher scores on benchmark datasets, it lacks the desired level of robustness. Additionally, once we have obtained the detection results for both players and the ball, we integrate them with the Hitter prediction generated by ViT. We designate the output generated by ViT as the hitter, while the remaining player is assigned the role of the defender. As the landing result of the ball is the desired information, we utilize the y-value of the bottom-right corner of hitter’s bounding box to represent the ball’s landing y-value. The landing coordinate of the ball is indicated by the black cross symbol depicted in Figure 5. Next, we use the vertices of the detection boxes closest to the ball as the positions of the Hitter and the Defender for the AICUP competition, as illustrated by the light blue and orange rectangles in Figure 5.

The intention behind incorporating TrackNetV2 in this context is to compensate for the shortcomings of YOLO in detecting badminton. Nevertheless, there may still arise scenarios where both YOLOv5m and TrackNetV2 fail to detect the shuttlecock simultaneously. In that case, I will assign the coordinates of LandingX and LandingY as $(0,0)$.

YOLOv8 Jocher et al. (2023) is the latest version of YOLO by Ultralytics. Being at the forefront of advanced technology, YOLOv8 represents a SOTA model that builds upon the achievements of its predecessors. It incorporates innovative features and enhancements to deliver superior performance, versatility, and efficiency. Furthermore, YOLOv8 provides comprehensive support for a wide array of vision AI tasks, encompassing detection, segmentation, pose estimation, tracking, and classification. In the CodaLab competition, we adopt a different approach for determining the positions of the players. Instead of relying on the vertices of the detection boxes, we utilize the pose estimated by YOLOv8x-pose-p6 model to acquire the players’ foot positions with enhanced accuracy. The image with the updated method is depicted in Figure 6.

The scoring system is based on each individual video clip, with a maximum score of $1$ point per clip. The first step is to validate the number of shots within the rally video, with each row in the corresponding csv file representing one shot. If the predicted number of shots differs from the ground truth, the question will receive a score of $0$ points. However, if the number of shots is accurately predicted, a score of $0.1$ points will be awarded, and the evaluation will proceed to the next stage, which involves comparing the contents of the columns. In this stage, the maximum score for correctly matching column content is $0.9$ points.

When the number of shots is accurate for the test video, each shot is evaluated individually, and its score is determined based on the cumulative score of its columns. The average score of all the columns is then used as the shot’s content score. The evaluation process begins by validating the HitFrame column. If the prediction error exceeds $2$ frames compared to the true value, the shot is awarded $0$ points in terms of content and the scoring proceeds to the next shot. However, if the prediction error is within or equal to $2$ frames, a score of $0.1$ points is given, and the remaining columns within the same shot are compared.

• •

  Hitter: $0.1$ points if correct, $0$ points otherwise.

• •

  BallHeight: $0.1$ points if correct, $0$ points otherwise.

• •

  Landing: If the prediction error is not greater than $6$ pixels in the Euclidean distance, it is considered correct and is awarded $0.1$ points; otherwise, $0$ points.

• •

  HitterLocation: If the prediction error is not greater than $10$ pixels in the Euclidean distance, it is considered correct with $0.05$ points, otherwise $0$ points.

• •

  DefenderLocation: If the prediction error is not greater than $10$ pixels in the Euclidean distance, it is considered correct with $0.05$ points, otherwise $0$ points.

• •

  Backhand: $0.05$ points if correct, $0$ points otherwise.

• •

  RoundHead: $0.05$ points if correct, $0$ points otherwise.

• •

  BallType: $0.2$ points if correct, $0$ points otherwise.

• •

  Winner: $0.1$ points if correct, $0$ points otherwise. (Note: If it is not the last shot but is filled in, it will be considered incorrect.)

Assuming that there are R rally videos in the data set, and the $i$-th video has $S_{i}$ shots, the scoring formula is given below:

$$\frac{1}{R}\sum\limits_{i=1}^{R}1_{S_{i}=S_{i}^{pred}}(0.1+ASS_{i})$$

(1)

in which $ASS_{i}$ (the Average Shot Score of content of the $i$-th rally video) is given by:

$$ASS_{i}=\frac{1}{S_{i}}\sum\limits_{j=1}^{S_{i}}1_{|HitFrame_{j}-HitFrame_{j}^{pred}\leq 2}SS_{j}$$

(2)

with $SS_{j}$ (the $j$-th Shot Score) given by:

	$\displaystyle SS_{j}$	$\displaystyle=0.1+0.1\times 1_{Hitter_{j}=Hitter_{j}^{pred}}$		(3)
		$\displaystyle+0.1\times 1_{BallHeight_{j}=BallHeight_{j}^{pred}}$
		$\displaystyle+0.1\times 1_{||Landing_{j}-Landing_{j}^{pred}<6}$
		$\displaystyle+0.05\times 1_{||HitterLocation_{j}-HitterLocation_{j}^{pred}<10}$
		$\displaystyle+0.05\times 1_{||DefenderLocation_{j}-DefenderLocation_{j}^{pred}<10}$
		$\displaystyle+0.05\times 1_{Backhand_{j}=Backhand_{j}^{pred}}$
		$\displaystyle+0.05\times 1_{RoundHead_{j}=RoundHead_{j}^{pred}}$
		$\displaystyle+0.2\times 1_{BallType_{j}=BallType_{j}^{pred}}$
		$\displaystyle+0.1\times 1_{Winner_{j}=Winner_{j}^{pred}}$

To begin with, we will discussing the hyperparameters and performance of SwingNet, as this aspect determines the upper limit of the overall score. This is due to the fact that any errors in the detection of ShotSeq and HitFrame would render subsequent feature accuracy irrelevant, as they would not contribute to the score. Subsequently, we will delve into the hyperparameters and performance analysis of the ViT-B/16 and YOLOv5m models. Consequently, the contribution of each model to the score will be illustrated at the conclusion of this section. Note that since the official pretrained TrackNetV2 and YOLOv8x-pose-p6 models are employed directly for inference, no further details will be provided here.

In this section, we will primarily address four key points of discussion. The first one is the extensive GPU memory resources required when using the method proposed in this article for shuttlecock hitting event detection with SwingNet. The main reason is that the model reads in $64$ frames of video at once for evaluation. For example, if we input $64$ full-color images with resolution of $720p$, the model needs to accommodate a size of $64\times 24\times 1280\times 720/8/1000\approx 169$GB. Due to the limited $6$GB memory of my GPU, the maximum image resolution we can use for training and testing is only $180\times 180$. I believe that if there is sufficient GPU memory available, utilizing the method proposed in this article and increasing the image size could be a promising approach. Compared to analyzing the trajectory of the shuttlecock, this method is more intuitive and can overcome situations where the shuttlecock is obstructed by the player.

The second point is that, according to the rules of badminton, we can infer that the information in the Hitter column will alternate between $A$ and $B$ $(A\rightarrow B\rightarrow A\rightarrow B\cdots)$. This implies that there will never be more than two consecutive instances of either $A$ or $B$. Consequently, by predicting the first hitter, we can determine the order of each hitter in a video. It is crucial to emphasize that when employing the alternative showup method, the accuracy of the predicted first hitter becomes paramount. Any inaccuracies in this prediction may result in a complete reversal of the order for every hitter in the entire video.

The third point to consider is that by exclusively focusing on the hitter during the classification process, it is possible to enhance the performance of the classification problem. That is, instead of utilizing the entire image for classification as depicted in Figure 3, one can solely utilize the area occupied by the hitter. Implementing this method has the potential to decrease the defender’s propensity to confuse hitter’s information, such as RoundHead, Backhand, and BallType.

Lastly, it is worth noting that in this competition, several features were not utilized while training, such as LandingX, LandingY, HitterLocationX, HitterLocationY, DefenderLocationX, DefenderLocationY. Without fully utilizing the available information, I believe there will be a certain degree of misalignment. Henceforth, we will analyze the unused information with the aim of leveraging it to aid in both the training and inference stages.