A Survey of Semen Quality Evaluation in Microscopic Videos Using Computer Assisted Sperm Analysis

It is of great significance for human beings to have the next generation. However, due to the unhealthy lifestyle of many male, their sperm quality is usually no good enough for fertilizing their wife. According to the research of [46], one in six couples fail to have a baby. $30\%$ of them are caused by male infertility. Therefore, the sperm analysis based onComputer-Aided Sperm Analysis (CASA) is meaningful for human beings. Besides this, for the agricultural trade, a good fertility rate for livestock would be a huge economic gain, which means employing CASA in this field is feasible and significant.

At present, most infertile couples realize their desire to have the next generation through In-Vitro Fertilization (IVF). Especially, Intracytoplasmic Sperm Injection (ICSI) is an effective way to solve the problem of infertility. However, due to the problem that the embryologist and technicians cannot select a sound sperm [22], IVF always has a high failure rate of about 60$-$70$\%$.

According to research of the World Health Organization (WHO) [39], quality of sperm can be measured by their movement types. Some specific criteria designed by WHO in 2010 are employed for sperm motility classification, seen in Tab.1. The classification of sperm motility is mainly determined by the speed and direction of sperm motility.

Traditional method of sperm quality evalution relies on manually counting through a microscope by experts. To achieve a detection result with more objective, some well-designed standards to determining which category the counted sperm belong to are necessary [32]. In spite of this, the traditional method has some disadvantages, such as time consuming, highly subjective and low accuracy [54]. Considering these disadvantages of traditional method and the importance of sperm quality assessment, it is particularly meaningful to employ CASA in this field.

In [15], a review about “assessing a single (donor) sample by the use of five CASA systems” is introduced, where it reveals that errors arising through variations in operator expertise and sample handling had considerably more influence upon the data than the more subtle differences between computer systems. Similarly, the work of [58] compares the Sperm Quality Analyzer IIC variables with the CASA estimates. According to these two reviews, CASA technology has better objectivity and accuracy than the traditional manual testing method. The introduction of CASA techniques into clinical laboratories should therefore help to improve laboratory standardization and the implementation of quality control procedures. With the development of CASA technology, CASA estimates, which will be more mature and provide much help for quantitative diagnosis, CASA system will become one of the important means to diagnose infertility [21].

We have first consult relevant works of literature and reviews about sperm quality analysis, and we find that most of them have the problems of containning too few or too old works of literature, and do not summarize or discuss the technology of using CASA. In the work of [1], it only summarizes nine papers from 2003 to 2010, and only compares simple applications of image processing, but it does not discuss the technical difficulties of CASA. In the survey of [3], only six works of literature about sperm video segmentation and detection are mainly discussed, and no in-depth analysis of the detailed methods is given. To this end, we organize this survey paper to summarize about 20 related works on CASA from 1988 to 2017, covering an extensive time span and focusing on the techniques used in the CASA field. As shown in Fig.1, with the increase of infertility rate and the decline of sperm quality in recent years, people’s attention and expectations for computer-assisted sperm quality analysis also show an increasing trend.

The main steps for CASA include: Specimen preparation, sperm segmentation and detection (localization), multi-sperm tracking and data association, and calculation of motility parameters. Fig.2 illustrates the general processing flow of CASA. Among this flow chart, commonly used algorithms in sperm detection and tracking are respectively exhibited.

As shown in Fig. 2:

• $\cdot$

  The first step of CASA is video acquisition. Most of the research teams’ microscopic videos are provided by hospitals and research institutes. There are also a small number of groups whose experimental data are provided by volunteers.

• $\cdot$

  CASA system is an image-based research work. Therefore, the second step of the experiment is to frame the microscopic video so as to facilitate the subsequent detection and tracking of the experiment.

• $\cdot$

  The third step is to detect or locate sperm. The main methods of detection and location include: Thresholding based detection, Otsu methods, optimum binarization methods, semi-automatic methods and so on. Details will be covered in Section 3.

• $\cdot$

  The fourth step of CASA is to track the sperm after detection. There are many common tracking algorithms applied to this step, such as mean shift, optical flow algorithm, particle filter tracking algorithm, and so on. There are also tracking algorithms developed by research teams themselves. These tracking algorithms are described in detail in Section 4.

• $\cdot$

  The final step in CASA is to restore the images of the tracking results to video.

This paper is organized as follows: Section 2, some related knowledge about CASA is introduced. In Section 3, we discuss sperm segmentation and detection (localization) techniques, and show representative images that demonstrate their operation. In Section 4, we summarize sperm tracking algorithms. Section 5 analyzes and prospects the present situation and future of CASA. In Section 6, we close this survey with a brief conclusion.

In [6] (section 3), the objectives of the CASA system in each stage are described in detail: (1) A successive image of the sperm suspension is projected onto the imaging detector; (2) sperm cells within each frame are detected based on pixel intensity and light scattering; and (3) professional software is used for obtaining necessary information and outputting the final result. To achieve the above objectives, the CASA system generally consists of four parts: Mechanical stage, illumination and optical systems, image acquisition (capture) and software output measures [2].

Mechanical stage: the optical microscope is moved to a predetermined position in the plane and autofocus is realized longitudinally.

Illumination and optical systems: CASA generally uses broadband lighting within the visible spectrum (390-700 nm). Pulse lighting is also typical in some systems, which cannot only meet the needs of the solid-state camera sensor shutter high-speed operation, but also can reduce the damage of sperm exposed to ultraviolet radiation. In addition, fluorescent lighting is also used in some devices to detect sperm stained with fluorescent pigments, to observe sperm morphology and whether there is membrane damage.

Image acquisition (capture): The acquisition of microscopic sperm video is accomplished by solid-state camera sensors (CCD or CMOS) in the hardware system. The smoothness of the image after framing is determined by the running speed of the camera shutter, which in turn affects the distance and path of sperm moving between successive frames. These parameters directly impact the output of the system and determine the accuracy of the CASA system diagnosis. The faster the shutter runs, the higher the video frames, and the closer the sperm motion-path is to the real situation, thus making the diagnosis more accurate.

All the existing CASA systems employ professional software [2]. There are mang open-source, such as Wilson-Leedy and Ingermann [60]. This procedure mainly includes sperm cell detection, tracking, and sperm motility parameter calculation. Moreover the system outputs the results, which serve as the diagnostic criteria for sperm detection.

Sperm health is mainly determined by two aspects: (1) Sperm motility parameters. For example, the moving speed mentioned in Tab. 1; (2) Sperm morphology. Sperm morphology has a decisive influence on sperm motility and movement speed. Morphologically, the majority of abnormal sperm can be roughly divided into two types: Double-headed or multi-headed sperm and malformed sperm. The comparison between normal sperm and abnormal sperm is shown in Fig. 3, (a) normal sperm, (b) double-headed sperm, and (c) malformed sperm.

The microscopic video processing of sperm is a complicated and challenging process. In the microscopic video of sperm, sperm cells are usually colorless or transparent, so the color and texture information is basically invalid in target detection. In addition, each sperm cell has a highly similar morphology, which dramatically reduces the reliability of tracking algorithms in extracting information from morphology. At the same time, the video background is doped with many protein blocks or other impurities, which puts forward higher requirements for the early processing of video images. Less color information, similar morphology, and more impurities are the characteristics of sperm microscopic video. An example of a microscopic video (a frame) of sperm is shown in Fig.4.

In [19], the key idea of sperm detection is analysis the the influence of different threshold on sperm movement parameters. In this experiment, 650 levels of all 2000 gray levels are tested and analysed. In [30], a thresholding-based segmentation technique is applied to process the sperm micro-video images. Especially, the judgement of which object segmented is identified as sperm head is determined by its size and gray value. After that, the center position of sperm head is obtained by analysing all the coordinates in this region.

In [35], an automatic sperm detection algorithm based on two-step threshold method is proposed. Firstly, a whole semen image is classified into foreground region and background region by employing a Otsu algorithm in global image histogram. After that, for obtaining detection result with higher precision, a Otsu algorithm is applied on a small square box that contains a particle, instead of a whole image.

In [23], sperm head region of interest (SHROI) is producted from original sample. First, center of SHROI is locadetd at where mouse points on. After that, an Otsu adaptive threshold algorithm [40] is employed for SHROI binarization. Third, to improve the performance of final detection, a morphological close operation are applied as the post-processing. Finally, the processed SHROI is converting into binary image, where object is setted to one and background is setted to zero. After all the operations are done, centroid of sperm head is located by analysing outline of the sperm head.

In [1], the main processing of sperm detect consists of four steps. Firstly, the input image is binarized by employing Sobel operator. Then an operation of morphological dilation is applied for eliminating line-shape noise. Third, all the objects related to image boundary are deleted. At last, operation of morphological erosion is performed twice to smooth the detected object. An example of this work is shown in Fig.5.

In [55], a threshold segmentation algorithm and some morphological operations are combined to detect sperm. A threshold segmentation algorithm is first applied to distinguish between the sperm region and the background region. In addition, sperm regions with possible single sperm are specially detected in this step. After that, morphological operation is performed twice. One is morphological erosion to eliminate noise and other debris. The other is morphological dilation to smooth sperm regions detected and filling small holes. At last, all the candidate regions are screened based their area informations. Only those candidate regions which area are similar in size to the sperm head area are retained. In addition, for achieving higher accuracy, manual operation is applied to remove incorrectly screened regions.

In [29], to get the detection target, a SHROI is first generated from initial image. The position of SHROI center depends on where the mouse points to. After obtaining a SHROI, Otsu adaptive threshold method is employed for image binarization to highlight sperms in SHROI. At last, edges of detected sperms in SHROI are obtained and analysed.

In [28], two consecutive frames are subtracted to obtain the boundary of sperm for target detection. After that, to make the boundary of sperm more clear, a threshold method is employed for eliminating noise and image binarization.

In [10], sperm detection part mainly consists of four steps. First, correspondent compensated phase map is obtained from original image. Second, Otsu threshold algorithm is applied for generating binary image. After automatically selecting regions of interest, the position of sperm is located based on selected function. Result of proposed detection method is shown in Fig.6.

In [11], different algorithms are integrated for a sperm segmentation task, including grayscale conversion, background identification, background subtraction, binarization, and binary morphology approaches. First, an adaptive histogram equalization method is applied to enhance local contrast of input image. After that, for better processing, image enhanced is transformed into gray-scale image. Third, the background generated is respectively employed for all frames based on a subtraction operation to eliminate image noise. Maximum entropy algorithm achieves the best performance compared to other binarization methods in image binarization experiment. Therefore, maximum entropy algorithm is applied for image binarization at last. Fig.7 shows the processing result.

The processing steps of proposed method in [57] is illustrated in Fig.8. First of all, a Gaussian filter is performed several times for eliminating noise and optimizing edge. Second, Laplacian-of-Gaussian (LoG) (“Mexican-hat”) filter is applied for enhancing the contrast between the object and its surrounding background. Based on this step, a spot-enhancement image is obtained. After that, the processed image is binarized by employing Otsu threshold. At last, a series of morphological operations are performed for binary image to improve detection accuracy. In the final processed image, only objects with less than 5 pixels are considered as sperm heads. Experiment results on prepared video frames suggest that the proposed method yields a high accuracy of $95\%$.

In [26], an automatic threshold segmentation method is proposed for sperm detection. The key idea of proposed method is to set a threshold named $T$ in advance. The input image is then binarized based on threshold $T$. Pixels with a pixel value above $T$ are setted to one. Other pixels are setted to zero. Fig.9 exhibits a result based on this method.

A summary of detection (localization) works reviewed is given in Tab. 2. We can see that many teams adopt self-developed detection algorithms to locate sperm and get good experimental results. Methods based on threshold are widely applied in sperm detection. Compared with other methods, methods based on threshold are simpler and easier to understand. In addition, methods based on threshold consist of many famous methods, such as Otsu threshold, local adaptive threshold. These alternatives give wider space for detecting and make threshold-based algorithm suitable to many different detection challenges that present on sperm microscopic video.

In [1], first frame of prepared video is selected for setting a baseline. In addition, based on sperm detection processing, numbers of sperms in all frames are respectively calculated and stored. Considering that sperm has a comparative low speed, regions surrounding sperm are focused on. Therefore, it is not necessary to spend a lot of time and effort to analyse the entire image.At the beginning of sperm tracking, frames are divided into many regular regions based on centroid of objects. When objects detected have similar shape and size with reference object in first frame, It are regarded as sperm. Then, taking the centroid position of the first frame as the starting point, the distance of centroid movement is calculated.

In [3], a 4-dimensional model consisting of two spatial coordinates, one time coordinate and one optical flow vectors direction is proposed. Based on proposed model, the problem of incorrect path reconstruction caused by collision is potential to be solved. First, an filtering is applied for generating pixel set of detected sperm. Then, pixels detected are represented by optical flow vectors. After processing all frames based on above method, a mean shift clustering method is employed for clustering all features obtained . At last, the correspondence between above clusters and initial path is analysed. The experiment results are shown in Fig.14, and this reference calculates the moving speed of the sperm.

In [67], an automatic method combining mean shift and a particle filter is proposed for tracking sperm. Sperm tracking using particle filters alone has the disadvantage of being too inefficient. Sperm tracking using particle filters alone has the disadvantage of falling into local optimal mode easily. In addition, both two methods fail to achieve efficiently tracking for sperms with random movement. Therefore, by combining this two methods, the proposed method can achieve rapid and accurate sperm tracking. The results of this algorithm are compared with results of particle filter and mean shift respectively, seen in Fig.15.

In [55], a multi-target tracker by employing tracking method mentioned in [18] is proposed. Experiment shows that the proposed method is able to track all sperms appeared in a fixed region. In addition, with the assistance of operator, the tracking operation can be continued for the other one after processing a sperm video.

In [23, 29], an automatic sperm head tracking method is proposed. First step is locating sperm head based on analysing image contour. A $40\times 40$ pixels region is then creating to surround sperm head, center of which is at the same position with center of sperm head. At last, all frames in prepared video are performed by the same operations to achieve sperm tracking. In addition, to make the object always in the center of the monitor area, the distance between center of the monitor area and sperm head is calculated. The distance data obtained is then employed by Proportion Integral Differential (PID) controller, which is able to make sperm in the center by moving view field. Fig.16 illustrates the result of sperm tracking based on proposed method.

In [28], a novel tracking methods is proposed for automatically tracking multiple moving sperm heads in the field of view at the same time. First, a subtraction operation is performed between two consecutive frames. Based on this step, a rough contour of sperm is obtained, which is then optimized by employing a fixed threshold value of 60. As the sperm video is processed frame by frame, the new captured contour of moved sperm replaces the previously captured contour in each frame of the video while the old contour is disappeared over time. At last, with the corresponding parameter settings, each contour exists for 0.5 seconds, and the color of the contour gradually lightens over time. Therefore, according to the sperm head contour traces of different shades of color, the movement route of sperm is obtained. In addition, to reduce the influence of noise and Brownian motions, several morphological operations are applied. The sperm tracking result is exhibited in Fig.17.

In [36], specific sperms are first chosen by professional operator. Center of selected sperm’s head is labelled manually. After that, method mentioned in [52] is employed for finding out the region that surrounds sperm head. Based on this method, edge of sperm head can be efficiently detected and the minimum box that surrounds sperm head is generated.

In [57], a joint probabilistic data association filter (JPDAF) method is proposed based on Matlab platform. Compared to original PDAF, the novel one comprehensively considers all the event probabilities associated with sperm, which is NP-hard [7, 47]. A probabilityweighted sum is applied by JPDAF to update sperm path. In addition, Kusiak’s method [20] is employed in processing paths and measurements obtained to achieve faster processing. Moreover, Cox’s adaptation method [5] of Kusiak’s is employed to reduce the amount of computation and computation time by selecting most highly probable joint association events. Since sperm cells can exhibit abrupt maneuvers, each independent Kalman filter uses a continuous white noise acceleration (CWNA) target motion model with an adaptive process noise covariance. Fig.18 exhibits tracking results based on proposed method.

In [10], a lot of computation time and effort is saved by employing proximity criterion. Specifically, according to the position of sperm in the $n$-th frame, unreasonable regions in the $(n+1)$-th frame are not considered. The tracking result with the assistance of proximity criterion is shown in Fig.19. Although it appears from Fig.19(b) that the paths of the two sperm seem to cross. In fact, they are on different planes, seen in Fig.19(c).In addition, proximity criterion is employed in $Z$ plane to achieve a tracking result with higher reliability. Based on this idea, objects with low contrast of phase map can be tracked.

In [45], a particle filter algorithm is applied to track the sperm. 300 particles is prepared for tracking one sperm. In addition, special sperm is hard to separate with others only based on velocity data when collision happens. Therefore, watershed method is employed for separating the special one with other sperms when collision happens. The Fig.20 shows the results of the treatment of sperm collision.

In [61], a reliable method based on two frames in a row is proposed for tracking sperm in video. The two frames in a row is regarded as a set of sample. For each sample, a sub-frame surrounding sperm head is generated in the previous frame, which is then matched to the next frame. The movement of sperm head is obtained based on the matching step, which is employed to generate a sub-frame for the next frame. The whole processing steps consisting of two stages is based on block matching method. The first stage is finding out the most suitable region by applying a similarity criterion. After that, the second stage is selecting most suitable global rigid transformation for all correspondences gained above. Fig.21 shows some head tracking results.

The task of sperm tracking presents two major difficulties. For one thing, sperm movement is relatively rapid and random. Most sperm, on the other hand, are similar in size and shape. In [17], a sperm tracking method combining particle swarm optimization and smoothing stochastic approximate monte carlo is proposed. In addition, to save the effort of finding areas that contain sperm, a windowing method is employed.

In [26], an automatic method combining computer technology and image processing technology is designed for sperm tracking. One challenge faced by the sperm tracking task is to determine the position of the same object in multiple consecutive frames of images. A $k$-nearest neighbor ($k$-NN) [13] is finally selected to deal this challenge. An tracking result based on $k$-NN is shown in Fig.22.

In [61], a novel algorithm for tail tracking is proposed. Based on the proposed method, shapes of flagellar beat are able to be obtained. The key idea of proposed method is replacing flagellum with several circles. Fig.23 shows some flagellum tracking results.

From this chapter, we can see that even though the processing of microscopic video images is generally the same, there are various views and applications of the tracking algorithm for sperm movement trajectory in different research teams. A summary of tracking works reviewed is shown in Tab. 3. Among them are the distance formula algorithm based on mathematics, and the conventional optical flow algorithm, the classical particle filter algorithm. There is also a tracking algorithm developed by the research team. However, due to the different data adopted by each algorithm, there are differences in evaluation standards, and there is no unified evaluation standard for CASA problem in the world. Therefore, the universality of these algorithms also remains to be investigated. Even so, from the experimental results, each algorithm shows a good tracking effect.

Through the previous summary, we can find that in the target detection stage, most teams adopt more traditional image segmentation methods: threshold detection, Otsu, and Otsu adaptive threshold [9, 8]. Some research teams also choose graphic fitting and semi-automatic detection. However, in terms of the detection effect, no matter which detection method can obtain more accurate detection of sperm cells, each method has achieved satisfying experimental results. From this view, CASA technology has no great difficulty in sperm cell detection.

When it comes to tracking sperm motility, each team takes a different approach. There are clear and simple distance calculation methods, probabilistic tracking algorithms, and more traditional particle filtering algorithms. The relatively new shape extraction method has also achieved acceptable tracking effect in [61], but its effect depends partly on the image quality of the microscopic video. As a classical tracking technique, particle filter algorithm is also applied in the CASA system, but its tracking accuracy is also closely related to the number of particles. This leads to a drawback of particle filter algorithm, when the number of particles is enough, the calculation speed of the algorithm will be substantially reduced, at the same time, at the cost of a high requirement on the storage space of the system. This is an unavoidable problem for clinical devices. In [26], $k$-NN is combined with deep learning. The convolution method is employed for features extraction from microscopic images, which greatly improves the accuracy of $k$-NN classification. After that, the researcher team uses distance calculations to track the movement of the same sperm. Feature extraction and target classification based on deep learning can identify and calculate the same sperm target more accurately for the tracking algorithm of distance operation. From Fig. 22, we can also see that the tracking effect of this algorithm is more similar to the real sperm movement track.

With the development of computer related technology and the improvement of related tracking algorithm, CASA system will provide clearer images for sperm as well as make the sperm movement trajectory depicted by the CASA system closer and closer to the real sperm movement trajectory. There are many factors influencing the accuracy of CASA results. On a technical level, the main factors generally include: the optical system for capturing video, frame rate, algorithms (detection and tracking algorithms).

For the problem of sperm trajectory tracking, the methods adopted in the early stage of image processing are roughly the same, because most of them adopt the means of computer graphics to detect and locate sperm. However, there are many kinds of tracking algorithms for target tracking. This leads to technical limitations for CASA, which focus on issues related to target tracking and fundamentally different ways of defining motion. Because of this, the CASA system has not been widely used in human medical clinical diagnosis. Relative to the original manual detection, CASA has some uncertainties. These factors may cause different systems to have different evaluation results for the same sperm sample. Nevertheless, the CASA system has been widely used in animal reproduction experiments and has achieved success. In [49], it is attributed to the “biological” reason, and the main influencing factors are compared in the article, see in Fig. 24.

At present, the feature extraction of sperm micro video mainly adopts static features and dynamic features. However, due to the similarity of sperm cells, these two characteristics cannot be applied well. Our research team adopts a new feature extraction method, foldover feature [26], a behaviour description that helps analyse objects with weak visual information. The foldover feature is of great significance in analysing dynamic objects in microscopic videos. Foldover is defined as followning: Each frame of an object’s motion is superimposed on the same spatial plane in the space-time order of the motion, the result of the superposition is the foldover of the object’s motion. Foldover of an object contains temporal information, spatial information, behaviour features and static features. The flow chart of the foldover feature algorithm is presented in Fig.25.

Although the evaluation criteria of the CASA system are not unified, from the perspective of sperm movement, the movement of each sperm is a transfer of the position of the center of mass. And there is a uniform definition of the value of movement. In [2], some measured value are listed, seen in Fig. 26. For example, curvilinear velocity [VCL], average path velocity [VAP], straight line velocity [VSL], amplitude of lateral head displacement [ALH], linearity of the curvilinear path [LIN], straightness of the average path [STR], and beat-cross frequency [BCF]. These values are essential for assessing sperm motility and can be used to rank levels of sperm health. Then, the percentage of motile sperm can be obtained. However, these motion values are rarely mentioned in the references above, and researchers focus more on whether the algorithm is accurate enough for sperm tracking. This may also be one of the limitations that prevent the CASA system from being used primarily for human reproductive health diagnosis.

Our review shows that the algorithms and technologies used by CASA are mostly classical methods, and not much application of the emerging algorithms. Therefore, we consulted some relevant technical literature of new algorithms and summarized some new methods that might be applied in the field of the CASA. In this way, the performance indexes of CASA system can be improved and the future development of CASA can be helped.

With advances in computer and Internet technology, the CASA system has great potential as a research tool in reproductive toxicology, animal production and human clinical analysis. Clearly, CASA needs to be rigorously tested to meet the standards of biological research and human reproductive medicine. In addition, CASA should pay more attention to sperm quality tests. Nowadays, male reproductive health diagnosis is highly subjective, so objective quantitative criteria are obviously urgently needed. Because this is more conducive to the quantification of sperm health analysis. It is not just looking at the motion tracking of sperm cells. CASA technology imperatively needs to improve the analysis of sperm motility, and incorporate it into the relevant experimental research. At the same time, the establishment of a large database with data analysis capability is also of great significance for the formulation of the diagnostic gold standard. These will improve the clinical relevance of semen analysis. More broadly, a stable and reliable CASA system will have a huge positive impact on natural life, wildlife conservation and animal husbandry.