Fish Tracking Challenge 2024:
A Multi-Object Tracking Competition with Sweetfish Schooling Data

Collective animal behaviors are teeming with life and intricate behavioral patterns. Fish schooling behavior offers a unique window into understanding animal navigation in water. For ethologists, ecologists, and mathematical and theoretical biologists, decoding these patterns is important. However, automatically tracking the movement of fishes, especially when in schools, introduces many challenges.

By developing advanced tracking platform [18, 15, 13] and models (e.g., [19, 20]), researchers can uncover the intricacies of aquatic movement and significantly advance this field. Originally, observation relied on the human eye [17], but recent technological innovations have fostered the increase of observational methodologies employing digital tools [7]. Utilizing digital video cameras facilitates objective and comprehensive observation, surpassing human visual capabilities, enabling simultaneous observation over wide areas [5]. Furthermore, the utilization of various recording devices such as night vision cameras, thermography cameras, sonar cameras, super slow-motion cameras, and drone cameras enables the observation of phenomena imperceptible to human visual inspection [6]. It is anticipated that observation methodologies leveraging digital equipment will continue to expand in the future.

The primary aim of this study is to advance the understanding and analysis of collective animal behavior in aquatic environments through the development and application of innovative multi-object tracking (MOT) models. By focusing on the intricate behaviors of schooling sweetfish, the study seeks to address and overcome the challenges associated with accurately monitoring and analyzing the movement and interaction patterns of aquatic animals in groups. The Fish Tracking Challenge 2024 (https://ftc-2024.github.io/), utilizing the comprehensive SweetFish dataset, provides a unique platform for researchers and technologists to develop, test, and refine advanced tracking algorithms capable of high-fidelity monitoring of multiple fish simultaneously.

This endeavor is not only important for the fields of ethology, ecology, and bio-navigation but also sets a precedent for interdisciplinary collaboration in the pursuit of understanding complex biological systems. The competition’s emphasis on leveraging video data and bounding box annotations to foster innovation in automatic detection and tracking algorithms aims to catalyze breakthroughs in how we approach the study of collective animal behavior. Ultimately, the study’s purpose is to enhance our scientific understanding of aquatic animal movements, contributing to broader applications in environmental conservation, sustainable fisheries management, and the development of autonomous navigation systems inspired by biological systems.

The remainder of this paper is organized as follows. First, in Section 2, we describe our Sweetfish dataset used in the competition. Next, we describe our baseline and competition winners’ methods in Section 3. We then present competition results in Section 4, and conclude this paper in Section 5.

In this competition, the dataset in the previous work [14] was used. The ayu or sweetfish (Plecoglossus altivelis) was collected, which are widely farmed throughout Japan. Juvenile ayus (approximately 7–14 cm in body length) shows typical schooling behavior. The experimental arena comprised a 3 $\times$ 3 $m^{2}$ shallow white tank. The water depth was approximately 15 cm (i.e., the schools were approximately two-dimensional). In the competition dataset, the spatial resolution of video was 2456 $\times$ 2048 pixels and a temporal resolution was 15 frames per second. The study [14] was conducted in strict accordance with the recommendations of the Guide for the Care and Use of Laboratory Animals from the National Institute of Health. The study protocol was approved by the Committee on the Ethics of Animal Experiments at University of Tsukuba (Permit Number: 14-386). All efforts were made to minimize animal suffering.

For each frame, the center point of each sweetfish was annotated. To adapt the dataset for the MOT task, the average bounding box size was calculated and applied to all annotations. The dataset was split into training, development, and test sets to facilitate model evaluation and generalization. The training set is used to train the model (i.e., bounding boxes are given), the development set is used to fine-tune the hyperparameters and to confirm the submission results (i.e., bounding boxes are given), and the test set is used to evaluate the final model performance (bounding boxes are not given). In total, the sweetfish dataset consists of 165,150 annotated bounding boxes of 10 sweetfishes. The dataset is divided into training (5 min 33 sec), development (1 min 15 sec), and test (11 min 33 sec) sets, with a total duration of 18 min 21 sec.

The goal of this challenge is accurate tracking of 10 sweetfishes. Performance of the models are be evaluated based on HOTA (Higher Order Tracking Accuracy) score, which is a holistic and popular score in MOT. HOTA is designed to overcome many of the limitations of previous metrics [11]. HOTA consists of detection accurary (DetA), localization accuracy (LocA), and association accuracy (AssA), the metrics combine the evaluation of detection accuracy, tracking accuracy, and false positives (FPs). HOTA finds a better balance between these two extremes by equally weighting both detection and association, while allowing analysis of each component separately with the detection accuracy and association accuracy sub-scores. In the competition results, the number of ID swithes (IDs) and false negatives (FNs) are also indicated for a reference.

As a condition to being awarded prizes, top-3 winners fulfilled the following obligations. After the final submission deadline, they submited their code so that the organizers can check for cheating. And they submited short report papers that describe the awarded methodologies [16, 9, 8].

In this section, we describe the methodologies by us (baseline method) and by the top participants of the Fish Tracking Challenge 2024, each utilizing the YOLOv8 object detector [10] and various tracking algorithms to address the complex task of sweetfish tracking. Here, we describe the methodologies from the baseline approach to the strategies employed by the first through third-place winners.

Tracking performances of the baseline and top-3 methods on the test dataset submitted the competition system are shown in Table 1. In summary, the baseline method achieves the best overall tracking performance. The 1st place method [16] based on HOTA score has the lowest number of IDs including baseline and the best scores among participants. The 2nd place method [9] shows better AssA than the 3rd place method [8]. The 3rd place method has better performances in the number of IDs, DetA, FNs, and FPs than the 2nd place method. These results suggest that more effort in adjusting detector and tracker hyperparameters, rather than correcting IDs, may result in a significant improvement in overall tracking ability.

Next, we briefly discuss the results of each method. In the 1st place method [16], the diversity of solutions in evolutionary computation may be limited because the data size of validation dataset was too small for the exploration by evolutionary computation. The detection was more robust than tracking and appearance change by wave on the surface of the water degraded the deep-learning-based tracking performance. In the 2nd place method [9], they found that ByteTrack [20] had a slightly better performance than BoTSORT [1]. Although ByteTrack is designed to handle occlusion, as the video progresses, they reported that the ID switches lead to a large amount of incorrect associations. They also considered that due to the direct usage from Ultralytics package [10], their detector failed to detect the fish while ground truth for them exist for many times, leading a high number of false negatives. They achieved the score increase by performing a post-processing technique, which includes merging and interpolation. Regarding the 3rd place method [8], for the occlusion problem, they can solve part of the ID switch problem through the “one-to-many” method, which is assigning a detection bounding box to two or more trajectories. For the problem of wave in fish detection, they skip the creation of additional trajectories by modifying the SORT code to keep the number of trajectories at 10, which can reduce certain missed detection and wrong detection problems.

In this paper, we introduced Fish Tracking Challenge 2024, a multi-object tracking competition focused on the schooling sweetfish. This paper outlines the competition’s objectives, the SweetFish dataset, and the methods of baseline and participants. The challenge emphasize the importance of multi-object tracking for discovering the dynamics of collective animal behavior, with the potential to significantly advance scientific understanding in the above fields. For future perspectives, to improve accuracy and robustness in the MOT task, exploring more sophisticated deep learning architectures and incorporating domain knowledges into the tracking models are considered. Regarding the extension of the current task, 3D tracking using multiple cameras and real-world aquaculture or ecological research settings can be expected.

This work is supported by JSPS KAKENHI under Grant Numbers 21H05300 and 21H05302. We would like to thank Dr. Naoya Yoshimura at Osaka University, Atom Scott at Nagoya University, and Alex Hoi-Hang Chan at University of Konstanz for their support in holding the competition.