Semi-supervised Cell Detection in Time-lapse Images Using Temporal Consistency

Non-invasive imaging techniques such as phase-contrast and differential interference contrast microscopy can capture images of cells without staining them. These techniques have been widely used for the long-term monitoring of living cells, in which hundreds of cells are captured as time-lapse images at short time intervals over days. Cell detection that detects the approximate positions of cell centroids from such microscopy images is a fundamental task for biomedical research [2, 8, 4, 32, 15, 17, 20, 8, 19, 5]. Time-lapse images include hundreds of cells, so manual analysis is time consuming. Therefore, there is significant demand for automated cell-detection tools.

Traditionally, image processing-based methods such as thresholding [16], level sets [24] have been proposed for cell detection [2, 3, 31, 27, 26]. Recently, convolutional neural network-based detection methods have performed well with a large amount of labeled data [4, 32, 15, 17, 20, 5]. For cell detection, heatmap-based methods have achieved promising results [15, 17, 20]. However, these methods require a certain amount of annotation for each imaging condition (e.g., type of microscopy, type of cell, and growth conditions). Preparing the necessary amount of annotation for each condition is a time-consuming and labor-intensive task.

To address this annotation problem, semi-supervised object-detection methods have been proposed mainly for bounding-box-based detection [7, 22, 21, 13, 28, 29, 1, 13, 11], and a few methods have been proposed based-on consistency learning [14, 6]. For example, Moskvyak et al. have proposed a consistency loss to train the network so that it consistently estimates the heatmap for data augmented by shifting or rotating [14]. These consistency-based methods assume that the labeled data are randomly sampled, i.e. labeled and unlabeled data have a similar distribution. However, this assumption often does not hold for time-lapse images of cell. In the time-lapse images, the cell density changes due to cell division, or the characteristics of cells may change due to the cultured condition (Fig. 1). As a result, the appearances of cells are different between the early frames and the later frames. Using consistency loss with an image as the label and the entire sequence as unlabeled data shows that the loss has an adversarial effect on training and does not work for the test data.

In this paper, we propose a semi-supervised cell-detection method that utilizes one time-lapse sequence, in which one image is labeled data and the other images are unlabeled data. Our method can improve the performance of the detection network by finding reliable detection positions from the unlabeled data using the labeled image as a clue. First, we train a detection network $f_{d}$ with a labeled image. As shown in Fig. 1, if we capture time-lapse images with short intervals, cells in frames close to the labeled frame have a similar appearance to the labeled-frame cells because the appearances of cells change gradually. Thus, the trained network can perform an accurate estimation for these frames close to the labeled image. In contrast, the estimation for the later frames often does not perform well since the cell appearance changes from the labeled frame due to the increase of the density, and stimulation from culture conditions. On the basis of this observation, we track cells using estimated results from the labeled frame to the further frames, and the consistently tracked cells are considered to be the reliable estimations. We use tracking results as pseudo-labels for retraining the detection CNN. By iteratively performing this process, our method can improve the detection CNN. Our main contributions are summarized as follows:

• •

  We propose a semi-supervised cell-detection method that can effectively train a detection network using a time-lapse sequence that contains one labeled image with the remaining images unlabeled. We improve the network by gradually adding pseudo-labels from the nearby frames.

• •

  We propose a pseudo labeling method for heat map-based cell detection by using tracking. We generate pseudo-heatmaps from high-confidence detection results that selected by tracking.

• •

  We demonstrated the effectiveness of our method for seven different conditions and demonstrated that our method can improve the detection network with one labeled image for various conditions.

An overview of the proposed method is shown in Fig. 2. Given one sequence $\mathcal{X}=\{x_{t}\}_{t=1}^{T}$, in which there is one labeled image $\mathcal{X}_{l}=\{(x_{l},y_{l})\}$ and the other images are unlabeled, our method improves the detection network $f_{d}$. $T$ is the number of frames in the sequence. (1) The cell-detection network $f_{d}$ is trained with a labeled image $\mathcal{X}_{l}$. (2) The prediction results $\mathcal{\hat{Y}}$ are obtained by the trained detection network $f_{d}$ from the whole sequence $\mathcal{X}$. (3) The detection results are tracked from the labeled frame $l$ to the frames far from it. (4) We generate pseudo-heatmaps $y^{p}_{t}$ and masks $M_{t}$ for each frame based-on the tracking result, and we get pseudo labeled images $\mathcal{X}_{p}=\{(x_{t},y_{t}^{p},m_{t})\}_{t=1}^{b}$. The mask $M_{t}$ is used on the next training step for mitigating the effect of untracked cells and mitosis cells. (5) The network $f_{d}$ is trained with generated pseudo labels $\mathcal{X}_{p}$. We improve performance by iteratively performing pseudo-labeling and learning.

The detection points of successive frames are associated by using a one-by-one matching [9], which optimizes the assignment among detected points in successive frames. The association is performed bi-directions, from the labeled frame to far frames. To avoid selecting the unconfident results, if the distance between the associated positions is small enough, we associate these. The right images in Fig. 2 show the examples of the estimated heat-maps $\hat{\mathcal{Y}}$ and the tracking results. We can observe that the heat-maps of frames close to the frame $l$, were more accurately estimated compared to the results of later frames. Thus, the number of successfully tracked cells from $l$ gradually decreases with far from $l$. If the pseudo-heatmaps contain many unreliable regions, which may be cell regions or background, these affect re-training. Therefore, we generate pseudo-heatmaps at the frames that the ratio of the tracked positions is larger than a threshold $\alpha$. The range of the tracked frames is defined as $[a,b]$.

Next, we generate the pseudo-heatmaps from the tracking results in $[a,b]$. We generate the set of the pseudo-heat-maps $\{y_{t}^{p}\}_{t=a}^{b}$ using the positions of the tracked cells $\{\vec{p}_{t}^{tr}\}_{t=a}^{b}$ in the same manner to the supervised heatmap generation [15]. These frames still contain few unreliable regions, i.e., the regions on un-tracked cells. To mitigate the affect from the un-tracked cell regions, we re-train the detection network using the masked loss that ignores the masked regions in training. There are two types of unreliable regions.

The first is a region that is detected but is not tracked. Most of this type’s region is the region of a daughter cell that newly appears by cell mitosis after frame $l$. Since cells monotonically increase with time by mitosis, we only consider this type of unreliable region at $t>l$. These regions can be easily defined using an unassociated detected positions $\{\vec{p}^{o}_{t}\}$. We define this region as $R(\vec{p}_{t}^{o})$ that is a set of pixels within radius $\beta$ from $\{\vec{p}^{o}_{t}\}$.

The second is a region that is not detected but there is a cell (miss detection). If detected points in a certain region are continuously tracked from frame $l$ until the previous frame $t_{ut}$ i.e., it is possible that miss-detection occurred at $t_{ut}+1$. To define the second regions, we use the position and timing when a track was terminated based on the tracking results, in which a terminate point and the frame are denoted as $\{\vec{p}^{ut}_{t^{ut}}\}$. Since cells move slowly, the positions of the un-tracked cell in the later/earlier frames can be roughly predicted by random work from the terminated position and time. A region at $t$ is defined as $R(\vec{p}^{ut}_{t^{ut}},t)$ that is a set of pixels within radius $\beta+||t^{ut}-t||$. The mask is defined using these unreliable regions as follows:

We train $f_{d}$ with $\mathcal{X}_{p}=\{(x_{t},y_{t},M_{t})\}_{t=a}^{b}$. When we train $f_{d}$ with $\mathcal{X}_{p}$, we use following loss function:

where $\vec{i}$ is a coordinate, and $N$ is the number of pseudo-labels. We repeat Pseudo-labeling and re-training the network are iteratively performed until we reach $\gamma$ iterations.

We compared our method with 3 conventional methods using one labeled image or unlabeled: Baseline, in which Nishimura’s method [15] was trained by one labeled image; Vicar [27], which is an image processing-based method by combining preconditioning of Yin [33] and distance transform [23]; Moskvyak [14], which is a consistency based semi-supervised detection method. In addition, to clarify how the number of the labeled data affect to the detection performance, we also compared several methods using additional labeled data: Supervised, in which Nishimura’s method [15] was trained by fully labeled image ; Moskvyak half, in which the first half of the sequence was additionally labeled and used for training.

Tab. 4 shows a quantitative evaluation of the F1 score on Cell Tracking Challenge. Our method achieves the best performance among the methods that trained with one labeled image. Since the time-lapse images gradually change the appearance of the image, Moskvyak’s performance worsens performance. Even if Moskvyak used 50 % of labeled data, the performance decreases from the baseline on DIC-C2DH-HeLa and PhC-C2DL-PSC. It indicates that the previous method that assumes the labeled images are randomly sampled can not work on our target setting. In contrast, our method can improve the performance with one labeled image. It can observe that our method is suitable for time-lapse sequences and effectively improves performance in a semi-supervised manner.

Fig. 4 shows the average F1 score of the three datasets for each frame in the training data. The horizontal axis indicates the F1 score, and the vertical axis indicates the frame. The performance of the F1 score is gradually decreasing from the twentieth frame. We observe that the performance of our method is improved compared to baseline by adding pseudo-labels from the close frames. Fig. 5 shows the example of result for iteration on DIC-C2DH-HeLa. As shown in Fig. 5, the heatmaps of our method become more clear than the baseline even in frames away from the labeled frame.

Table 2 shows the quantitative evaluation results with the F1 score on four cultured conditions. Our method achieves the best performance on whole-culture conditions. Even if the cell shape slightly changes depending on the culture condition, the detection performance can be improved by this method. It indicate that the proposed method is effective for various time-lapse images.

We proposed a semi-supervised cell detection method for time-lapse images in which there is one labeled image and the other images are unlabeled. Our method can improve detection network by adding pseudo labels that is selected by tracking from detection results. We demonstrated our method’s effectiveness on seven different conditions, and we demonstrated that our method can improve detection performance on various conditions.