TOD-CNN: An Effective Convolutional Neural Network for Tiny Object Detection in Sperm Videos

Traditional methods mainly include three types, which are threshold-based methods, shape fitting methods, and filtering methods. Threshold-based methods: Urbano et al. [14] use Gaussian filter to enhance the image, and then the image is binarized using the Otsu [32] threshold method, and the result is morphologically operated to determine the position of the sperm; Elseyed et al. [13] use several certain frames to generate the background information, then the background information is subtracted from the original image (to suppress noise). Finally, the Otsu threshold is applied to determine the position of the sperm. Shape fitting methods: Zhou et al. [33] use a rectangular area which is similar to the shape of the object (sperm) to fit the object, and then the position of the sperm is described by the parameters of the rectangle. Yang et al. [16] use an ellipse to approximate the sperm head, and the improved multiple birth and cut algorithm based on marked point processes [34] is used to detect and locate the head of the sperm through modelling. Filtering methods: Ravanfar et al. [35] select several suitable structural elements firstly, and then the operation based on Top-hat is used to filter the image sequence to achieve the purpose of separating sperm and other debris. Nurhadiyatna et al. [36] use the Gaussian Mixture Model (GMM) enhanced by the Hole Filling Algorithm as the probability density function to predict the probability of each pixel in the image belongs to the foreground and the background. The researchers found that the calculation amount of this method is significantly less than other methods.

The unsupervised learning method is the most used machine learning method. Berezansky et al. [37] use the Spatio-Temporal Segmentation to detect sperm by segmentation, integrating $k$-means, GMM, mean shift, and other segmentation methods. Shi et al. [38] use the optical capture method for sperm detection.

The YOLO series models [22, 23, 24, 25], RetinaNet [27], and SSD [26] are prominent representatives of one-stage object detectors. The one-stage object detectors are based on the idea of regression, which can directly output the final prediction results from the input images without generating suggested regions in advance. YOLO series models: They use Darknet as the backbone of the model to extract features from the image. The v1, v2, v3, v4 are successively proposed by improving the backbone network structure, improving the loss function, using batch normalization, feature pyramid network [48], spatial pyramid pooling network [49], and other optimization methods. RetinaNet: It uses ResNet [50] and the feature pyramid network as the backbone of the model, whose main contribution is proposing a focus loss function. The focus loss function solves the imbalance between the number of foreground and background categories in a single-stage object detector. SSD: It uses VGG16 [51] as the basic model and then adds a new convolutional layer based on VGG16 to obtain more feature maps for detection and generates the final prediction result by fusing the prediction results of 6 feature maps.

The models based on R-CNN series are classical two-stage object detectors. The R-CNN [18] generates proposal regions through the selective search algorithm. Then the features of the proposal regions are extracted by using CNN. Finally, the SVM classifier is used to predict the objects in each region and identify the category of the objects. The Fast R-CNN [19] no longer extracts features for each proposal region. The features of entire image is extracted using CNN, then each proposal region and corresponding features are mapped. Besides, the Fast R-CNN uses a multi-task loss function, allowing us to train the detector and bounding box regressor simultaneously. The Faster R-CNN [20] replaces the selective search algorithm with Region Proposal Network, which can help CNN to generate proposal regions and detect objects simultaneously.

YOLO series models solve the object detection task as a regression problem. The YOLO series models remove the step of generating the proposal region in the two-stage object detector and accelerate the detection process. YOLO-v3 [24] is the most popular model in the YOLO series models due to its excellent detection performance and speed.

The YOLO-v3 model mainly consists of four parts, which are preprocessing, backbone, neck, and head. The preprocessing: The $k$-means algorithm is used to cluster nine anchor boxes in the data set before training. The backbone: YOLO-v3 uses Darknet53 as the backbone network of the model. Darknet53 does not have maxpooling layers and fully connected layers. The fully convolutional network can change the size of the tensor by changing the strides of the convolutional kernel. In addition, Darknet53 follows the idea of ResNet and adds a residual module to the network to solve the vanishing gradient problem of the deep network. The neck: The neck of the YOLO-v3 model draws on the idea of the feature pyramid network [48] to enrich the information of the feature map. The head: The head of YOLO-v3 outputs 3 feature maps with different sizes and then detects large, medium, and small size objects of the three feature maps with three sizes.

ResNet [50] is one of the most widely used feature extraction CNNs due to its practical and straightforward structure. With the continuous deepening of CNN, the model’s performance cannot be continuously improved, and the accuracy may even decrease. However, ResNet proposes the Shortcut Connection structure to solve the problems above. The identity mapping operation and residual mapping operation are included in the Shortcut Connection structure. The identity mapping is to pass the current feature map backward through cross-layer transfer (when the dimension of feature map does not match, a $1\times 1$ convolution operation is used to adjust the dimension of feature map). Residual mapping is to pass the current feature map to the next layer after convolution operation. A Shortcut Connection structure contains one identity mapping operation and two or three residual mapping operations in general.

In Inception-v3 [52], to reduce the parameters and ensure the performance of the model, an operation that replaces $N\times N$ convolution kernels with $1\times N$ and $N\times 1$ convolution kernels is proposed. The receptive fields of $1\times N$ and $N\times 1$ convolution kernels and $N\times N$ convolution kernels are the same, where the former has less parameters than the latter. In addition, the Inception-v3 model can support multi-scale input, which can use convolution kernels with different sizes to perform convolution operations on the input images, and then the input feature maps can be connected to generate the final feature map.

VGG16 [51] model includes 13 convolutional layers, 3 fully connected layers, and 5 maxpooling layers. The most prominent feature of the VGG16 model is its simple structure. All convolutional layers use the same convolution kernel parameters, and all pooling layers use the same pooling kernel parameters. Although the VGG16 model has a simple structure, it has strong feature extraction capabilities.

The object detection task in the video is essentially based on image processing. Therefore, it is necessary to split the sperm microscopic video into continuous frames (single images). However, due to the movement of the lens during the sperm microscopic video shooting process, there are some blurred frames in the sperm microscopic video. After analysing the grayscale histogram of frames, there is an obvious difference between the grayscale distribution of the blurred frame and the normal frame. Therefore, the blurred frames can be solved by deleting images whose Otsu threshold is less than a certain threshold (from articial experience). In addition, TOD-CNN is an anchor-based object detection model. Therefore, the $k$-means algorithm is used to cluster a certain number (TOD-CNN uses six) of anchor boxes in data set to train the model.

A straight forward backbone structure with cross-layer concatenate operation is designed, which is shown in Fig. 3. However, in a fully convolutional network, as the structures of CNNs continue to deepen, the semantic information of the feature map becomes more and more abundant, while the location information of the feature map constantly decreases. As a result, the network can improve the classification performance but may reduce the positioning accuracy. Our work focuses on detecting tiny objects and accurate locating, which needs to maintain precise local information. Therefore, we enhance the transfer of location information (transferring shallow features to deep layers) through the following methods: First, we refer to the residual idea of ResNet, the Shortcut Connection structure provides the approach for transferring local information with a cross-layer add operation, which is used in TOD-CNN (as shown in Res (A, B, C) in Fig. 3); second, based on the straightforward backbone structure, a cross-layer concatenate operation is applied to enhance the transfer of local information (as shown in grey shaded part in Fig. 3).

The detailed design of TOD-CNN backbone is shown in Fig. 3, where the input size of the backbone is $416\times 416$, the yellow arrow indicates convolution operations with a kernel size of $3\times 3$ and stride of 1 (each use filtering with padding and followed by a Mish activation), the red arrow indicates convolution operations with a kernel size of $3\times 3$ and stride of 2 (each followed by a Mish activation), and the green arrow indicates convolution operations with a kernel size of $1\times 1$ and stride of 1 (each use filtering with padding and followed by a Mish activation).

In object detection model, the main purpose of model neck is to integrate feature informations extracted from model backbone. The neck structure of TOD-CNN is shown in Fig. 4. In fact, there are abnormal morphological sperms (very big) and some other impurities (such as bacteria, protein lumps and bubbles) in semen. These sperms and impurities are significantly different from normal sperm in size. Therefore, to collect multi-scale information, we have adopted spatial pyramid pooling operation [49] to integrate multi-scale information into TOD-CNN neck. In addition, due to the small sizes of tiny objects, the information of tiny objects might be easily lost in down-sampling process. In order to solve this problem, the feature fusion method is used in TOD-CNN neck, where the shallow and deep feature maps are fused by upsampling to avoid the loss of tiny object information.

The detailed design of TOD-CNN neck are shown in Fig. 4, where all convolution operations are with stride of 1 (each use filtering with padding ), and the detailed illustration of the kernel size and activation function is shown in Fig. 4. Finally, TOD-CNN neck outputs a feature map of size $($input size$/\ 8)\times($input size$/\ 8)\times 42$, where 42 is the number of anchor boxes$\ (6)\times 7$, because each anchor box needs to have 7 parameters: the relative center coordinates, the width and the high offset, class, and confidence, the details are explained in Section 3.2.4.

In the head of TOD-CNN, 6 bounding boxes are predicted for each cell in the output feature map. For each bounding box, 7 coordinates ($t_{x}$, $t_{y}$, $t_{w}$, $t_{h}$, $C$, $P_{0}$ and $P_{1}$) are predicted, so the dimension of the predicted result in Fig. 5 is 42. For each cell, the offset from the upper left corner of the image is assumed to be ($C_{x}$, $C_{y}$), and the width and height of the corresponding a priori box are $P_{w}$ and $P_{h}$. The calculation method of center coordinates ($b_{x}$ and $b_{y}$), width ($b_{w}$) and height ($b_{h}$) of predicted box is shown in Fig. 5. Multi-label classification is applied to predict the categories in each bounding box. Furthermore, due to the dense prediction method is applied to the head of TOD-CNN, the non-maximum value suppression method based on distance intersection over union [53] is used to remove bounding boxes with high overlap in the output results of the network.

A sperm microscopic video data set is released in our previous work [54] and it is used for the experiments of this paper. These sperm microscopic videos in the data set are obtained by a WLJY-9000 computer-aided sperm analysis system [55] under a $20\times$ objective lens and a $20\times$ electronic eyepiece. More than 278,000 objects are annotated in the data set: normal, needle-shaped, amorphous, cone-shaped, round, or multi-nucleated head sperms and impurities (such as bacteria, protein clumps, and bubbles). The object sizes range from approximately 5 to 50 $\mu$m². These objects are annotated by 14 reproductive doctors and biomedical scientists and verified by 6 reproductive doctors and biomedical scientists.

From 2017 to 2020, the collection and preparation of this data set took four years, including more than 278,000 annotated objects, as shown in Fig. 6. Furthermore, the data set contains some hard-to-detect objects, such as uncertain morphology sperm, low contrast sperm, and similar impurities (as show in Fig. 7), which greatly increases the difficulty of tiny object detection.

In this data set, Subset-A provides more than 125,000 objects with bounding box annotation and category information in 101 videos for tiny object detection task; Subset-B segments more than 26,000 sperms in 10 videos as ground truth for tiny object tracking task; Subset-C provides more than 125,000 independent images of sperms and impurities for tiny object classification task. Although Subset-C is not used in this work, it is still openly available to non-commercial scientific work.

We randomly divide the sperm microscopic video into training, validation, and test data sets at a ratio of 6:2:2. Therefore, we have 80 sperm microscopic videos and corresponding annotation information for training, and validation. The training set includes 2125 sperm microscopic images (77522 sperms and 2759 impurities), and validation set includes 668 sperm microscopic images (23173 sperms and 490 impurities). And we have 21 sperm microscopic videos for testing, the test set includes 829 sperm microscopic images (20706 sperms and 1230 impurities).

The experiment is conducted by Python 3.7.0 in Windows 10 operating system. The models we use in this paper are implemented by Keras 2.1.5 framework with Tensorflow 1.13.1 as the backend. Our experiment uses a workstation with Intel^(R) Core^(TM) i7-9700 CPU with 3.00GHz, 32GB RAM, and NVIDIA GEFORCE RTX 2080 8GB.

The purpose of object detection task is to find all objects of interest in the image. Therefore, this task can be regarded as a combination of positioning and classification tasks. Therefore, as the loss function of the network, we use the complete intersection over union [53] (CIoU) function (location loss function) and the binary cross-entropy function (confidence and classification loss function), and then minimize them by Adam optimizer. For other hyper parameters, when freezing part of the layer training and unfreezing all layers, the batch size is set to 16 and 4, the training is 50 and 100 epochs, and the learning rate is set to $1\times 10^{-3}$ and $1\times 10^{-4}$, respectively. Besides, the cosine annealing scheduler [56] is used to adjust the learning rate. Besides, when the loss value no longer drops, the training is terminated early.

In this part, we make a comparison between TOD-CNN and some state-of-the-art methods in terms of memory costs, training time, FPS, and detection performance.

To verify the reliability, stability and repeatability of TOD-CNN, we have performed five-fold cross-validation. The experimental results are shown in Table LABEL:Tab3, where the mean values ($\mu$) of the four evaluation metrics is higher than $89\%$ except for AP, and AP is higher than $86\%$. It can be seen that TOD-CNN has good performance and repeatability. The standard deviation (STDEV) of F1 is $1.02\%$, the STDEV of two of the four evaluation metrics are below $1.40\%$, and only the STDEV of Pre is slightly higher ($2.05\%$), showing that TOD-CNN is relatively stable and reliable.

The ultimate goal of sperm detection is to find the sperm trajectories and calculate the relevant parameters for clinical diagnosis. TOD-CNN and two models with better detection results (YOLO-v4 and SSD) are are compared for sperm tracking in Table 3. Based on the detection result of each model, we use the $k$NN algorithm to match sperms in adjacent video frames to the actual trajectories marked in Subset-B. The visualization results are shown in Fig. 10. We can observe that our tracking trajectories are very close to the actual ones, and the trajectory discontinuity or incorrect tracking is rarely occurred due to the stronger detection capability of TOD-CNN.

In addition, we calculate three important motility parameters of sperms on Subset-B, including the Straight Line Velocity (VSL), Curvilinear Velocity (VCL) and Average Path Velocity (VAP) [59] of actual trajectories, with TOD-CNN, SSD and Yolo-v4, respectively. Comparing with the actual trajectories, the error rates of VSL, VCL and VAP calculated with TOD-CNN (10.15%, 5.09% and 8.95%) are significantly lower than that of SSD (41.58%, 5.01% and 17.40%) and Yolo-v4 (12.73%, 36.12% and 19.65%). Based on VCL, VSL, and VAP, an experienced threshold value from a clinical doctor is set to determine whether a sperm is motile to calculate the corresponding progressive motility (PR). The error between PR obtained by TOD-CNN tracking results and doctors’ diagnosis results are all within 9%. The experimental result shows that our TOD-CNN can assist doctors in clinical work.

To conveniently use TOD-CNN to detect tiny objects in microscopic videos and images, we design a Python-based GUI (Fig. 11) that can help users to control the Intersection of Union (IoU) threshold and confidence according to their own needs to achieve the desired test performance. Besides, users can load Model Path to use their own setting/weights for tiny object detection. This GUI is compatible with videos (such as “.mp4” and “.avi”) and images (such as “.png” and “.jpg”) in various formats.