Unsupervised Data-Driven Nuclei Segmentation for Histology Images

Histology images provide strong cues to pathologists in cancer diagnosis and treatment. Automated nuclei instance-level segmentation for histology images provides not only the number, density and sizes of nuclei but also morphological features, such as the magnitude and the cytoplasmic ratio. This information facilitates cancer diagnosis and assessment of the tumor aggressiveness rate. Nuclei segmentation tasks can be conducted in a supervised or an unsupervised manner. For supervised methods, the annotation of high resolution histology images in pixel-level accuracy is a time-consuming job, being carried out by expertized physicians. Other identified challenges include the variability of cell appearance from different organs, unclear boundaries, as well as color and intensity variations in stained images from different laboratories. Furthermore, it is a subjective task and annotated labels tend to vary from one person to the other. All the above factors challenge the practical generalizability of supervised segmentation methods, as histology images become larger in their number and more diversified in content.

Earlier methods on nuclei segmentation were mainly unsupervised. They were based on thresholding [1, 2], mathematical morphology for robust feature extraction [3], or statistical modeling for segmentation and likelihood maximization for boundary refinement [4]. Another popular tool was the watershed algorithm, which was combined with various ways for extracting potential nuclei locations [5, 6]. Moreover, a work by Ali et al. [7] proposed an adaptive active contour mechanism that takes boundary- and region-based energy terms of the cell into account.

Recently, deep-learning-based (DL) methods [8, 9, 10, 11] have been applied to this problem. A wide variety of DL models are trained by human labeled data. In general, supervised methods provide significantly better performance than the unsupervised ones. The motivation of our research stems from the fact that nuclei instance labeling is a fairly laborious task, with reportedly highly subjective annotations and miss-labeling rate [12] that challenges the supervised solutions. Several studies have been made to mitigate the labeling cost. For example, Qu et al. [13] proposed a two-stage learning framework using coarse labels. Other researchers [14, 15] investigated the self-supervised DL method to reduce the number of required labeled data by exploiting the observation that nuclei size and texture can determine the magnification scale. Besides, there is a domain shift problem [16] arising from stain and nuclei variations in histology images of different organs, patients and acquisition protocols.

In spite of the high performance, DL-based solutions have their own shortcomings on top of the high labeling cost. First, they are perceived as a “black-box” approach. Interpretable solutions are highly desired in medical applications as they enable explainable decisions and the tools are more trustworthy in clinical use. Second, they bear a high computational cost in training and testing [17] because of a large number of parameters in order to achieve a certain performance.

In this work, we propose an unsupervised data-driven method to solve the nuclei segmentation problem. The solution consists of three modules applied to each image block of size $50\times 50$: 1) data-driven color transform for energy compaction and dimension reduction, 2) data-driven binarization, and 3) incorporation of geometric priors with morphological image processing. It is named “CBM” because of the first letter of the three modules – “Color transform”, “Binarization” and “Morphological processing”.

We conduct experiments on the MoNuSeg dataset to demonstrate the effectiveness of the proposed CBM method. It outperforms all other unsupervised approaches and stands in a competitive position with supervised ones based on the Aggregated Jaccard Index (AJI) performance metric. Thus, the proposed CBM method is an attractive solution in practice nuclei segmentation, since it yields comparable performance with state-of-the-art supervised methods while requiring no training labels.

Each histology test image of the experimenting dataset has a spatial resolution of $1000\times 1000$ pixels and each pixel has R, G and B three color channels. We partition each image into nonoverlapping blocks of size $50\times 50$ pixels and apply contrast enhancement to accentuate the boundaries around nuclei as a pre-processing step. Then, CBM processes each block independently with three modules as elaborated below.

The data from the 2018 MICCAI MoNuSeg Challenge [9] are used to evaluate the performance of our proposed CBM method. This is a popular dataset for existing work, providing also different testing configurations. For the validation purpose, we follow the data splitting as suggested in [9].

For performance benchmarking with other existing works, we report the results on the 6 histology images set from bladder, colon, stomach, referred as Test-2, as well as the test data from the challenge set (14 images from various organs) in Table 2, where the commonly used AJI metric [9] is adopted. The AJI metric is more accurate for instance-level segmentation tasks by taking both the nuclei level detection and pixel-level error performance into account.

As shown in the table, our method outperforms all other unsupervised ones by a large margin in either testing scenarios. Notably, the gap is large even for the DL-based unsupervised approach denoted by “self-supervised” [14] in the table. As compared with supervised methods, CBM stands at a competitive position. It achieves similar performance with sophisticated models such as BES-Net [20]. It is also worthwhile to emphasize that most of the supervised methods use expensive models as a backbone architecture (in terms of the number of its parameters) such as Res-Net or Dense-Net. Instead, CBM is a parameter-free method with simple components. Its computational complexity is significantly lower.

It is also evident from Table 2 that supervision may help to reach higher performance. Nevertheless, it is challenging for most DL supervised methods to generalize well in this problem, especially when the training data are in paucity. For instance, the current state-of-the-art SSL method achieves the reported performance using the full training set. According to their weakly supervision analysis [15], SSL achieves comparable performance with CBM when trained with roughly 50% of the data.

To illustrate the energy compaction capability of the PQR channel decomposition, we compare the AJI performance for the $P$ channel and the $L$ channel (in the LAB color space) in Table 3. The advantage of using the $P$ channel is clearly demonstrated in the table.

Nuclei segmentation in histology images is a demanding and prone to errors task for physicians, and its automation is of high importance for cancer assessment. The proposed CBM method offers a promising lightweight unsupervised direction for nuclei segmentation, requiring no labeled data and no parameter cost. It addressed the problem based on data-driven color conversion and binarization, as well as a morphological module that takes into account priors about the nuclei shape and size. In the future, we would like to boost the segmentation performance furthermore by exploring weakly or self supervision.