A three in one bottom-up framework for simultaneous semantic segmentation, instance segmentation and classification of multi-organ nuclei in digital cancer histology

Keypoint-based approaches regress the nuclei locality by using pre-defined key points to the nuclei. The keypoint approaches are classified into two categories, i.e., group-based and group-free key point approaches. Group-based keypoint approaches produce key points for each nucleus and then group them based on post-processing methods to form bounding boxes. CornerNet [14], CentripetalNet [15], and RepPoints [16] are examples of group-based keypoint approaches. Group-based keypoint approaches have similar drawbacks to traditional image-processing methods since they depend on traditional post-processing methods. Contrary to the group-based keypoint approaches, the group-free keypoint approaches detect the nuclei center directly. Group-free keypoint approaches include CenterNet [17], SaccadeNet [18], and PointNu-Net [4]. The drawback of the keypoint-based methods is that they achieve lower Segmentation Quality (SQ) since they output the high confidence instances only, thus reducing the classification performance. Keypoint-based methods produce an enormous amount of incorrect bounding boxes [17]. Furthermore, keypoint-based methods also have reduced inference speed [19].

Top-down approaches produce candidate regions and use non-maximum suppression tools to obtain bounding boxes. Then segmentation and classification of nuclei are independently performed. Examples of top-down approaches include Mask-RCNN [20], YOLACT [21], CenterMask [22], PolarMask [23], TensorMask [24], and MaskLab [25]. Although the top-down approaches are commonly used for generic applications, these methods are rarely used for segmentation and classification for many reasons. One reason is that these approaches have shown lower performance for segmenting nuclei boundaries due to densely populated and overlapped nuclei. These approaches may recommend tens of overlapped bounding boxes in a small region of a histology image. Furthermore, due to non-maximum suppression, a bounding box may not cover the whole nuclei.

Bottom-up approaches produce high-resolution semantic segmentation and then utilize post-processing methods to group and clean the segmented and classified nuclei. Some top-performing bottom-up approaches include, [26], DIST [6], CIA-Net [7], Triple U-Net [8], HoVer-Net [5], and [11]. Kumar et al. [26] predict the background, nuclei, and edges of the nuclei and then cluster the pixels into instances based on region growth using seeds. CIA-Net [7] uses contour supervision to generate edges. DIST [6] regresses distance maps to group pixels. HoVer-Net [5] uses separate vertical and horizontal distance maps of the nuclei pixels from their centers to achieve simultaneous instance segmentation and classification. Although these bottom-up approaches are commonly used for nuclei segmentation and classification due to their superior performance, they still possess drawbacks. These include problems like variation in tissue, poor staining quality, variable nuclear density (nuclei overlapping), and nuclei class imbalance. These methods still possess a significant performance gap in extreme clinical conditions.

Encoder-Decoder-based networks have improved segmentation performance in digital histology compared to the rest of the networks. However, these networks possess limitations. Usually, the encoder and decoder features are directly fused. Some of the features may possess redundant information, thus increasing complexity and may even reduce performance. Digital histology is H&E stained, where most information lies in the channel. Furthermore, some information is also spatially present; for example, the inflammatory nuclei usually surround the tumor nuclei. Therefore considering these features may increase the network performance. The second problem is the design of the encoder. The encoder is comprised of consecutive convolution layers, which produce discontinuity between lower-level and higher-level features. It also introduces the vanishing gradient problem. Inspired by these observations, we introduce spatial and channel attention mechanisms. We place our attention mechanism between the encoder and decoder as an intermediate layer. We call it the attention decoder block. Transposed convolutions follow the attention decoder blocks. We have replaced simple convolutions blocks with ResNet blocks in the encoder. Fig. 2 shows the three-head network and building blocks in detail.

The three-head network has 16 filters at the first stage (represented by ‘F‘ in Fig. 2), and then at each subsequent stage of the encoder, the number of filters is doubled. At each subsequent decoder stage, the number of filters is halved.

We use a controlled watershed algorithm to get nuclei instance segmentation. The instance segmentation framework is shown in Fig. 3. First, we remove the edge proposals having less than ten percent probability. This limit is chosen such that the framework provides optimal performance. This step removes some of the false edges. We subtract these edge proposals from semantic segmentation maps in the next step. This separates the connected nuclei and gives us the sure foreground region (foreground markers) of the nuclei. The background markers in the semantic segmentation image are shown in black color. Ideally, we do not want these background markers to be too close to the edges of the nuclei. This is done by computing the watershed transform of the Euclidean distance transform of sure foreground markers and then identify for the watershed ridgelines. We combine all these, i.e., sure foreground, ridgelines, and background. Finally, we apply a marker-controlled watershed algorithm to get instance nuclei.

We take the previously segmented instances for classification and extract the corresponding region in the pixel-wise class image obtained from the three-head network. Then we separately count the pixels of each class within the extracted region. Finally we assign the selected instance to the region’s class with the highest frequency. Mathematically, the process is represented as Eq. 7.

where, $y_{c}^{ins}$ is the class ($c$) assigned to the instance ($ins$), $x_{cn}\left(i,j\right)$ pixel class at row $i$, column $j$ predicted by the three head network, and $n$ is the total number of classes. An illustration of the pixel grouping is shown in Fig. 4.

The proposed three head framework have three sets of losses ($l_{a}$, $l_{b}$, and $l_{c}$) and their respective weights ($\lambda_{a}$, $\lambda_{b}$, and $\lambda_{c}$) , combined under 8,

For semantic segmentation and edge proposal, we retain our previous loss function [27, 28], since it has proven to be the most productive for semantic segmentation. This loss function is represented by 9,

We have used this specific loss function for semantic segmentation and edge proposals since it has shown slightly improved performance compared to the other widely used loss functions. For the classification head, we have used categorical cross-entropy loss, represented by 10,

where, $p$ are the predictions, $t$ are the targets, $c$ denotes the data point, $d$ denotes the nuclei class, and $w_{d}$ denotes individual class weights. It is to be noted that we have used separate weights ($w_{d}$) within the categorical cross-entropy as well in order to tackle the high nuclear category imbalance. These weights are assigned based on the percentage of the nuclear category, reported by PanNuke [9]. The other weights, i.e., $\lambda_{a}$, $\lambda_{b}$, and $\lambda_{c}$ are, specifically, set to 1, 5, and 4, respectively, for optimal performance. The improvement in performance due to the addition of these weights is verified and reported in the ablation study (Section 5.2) of this work.

PanNuke is the largest publically available cross-tissue nuclei cancer histology dataset with thousands of images. PanNuke dataset is obtained from combining four different datasets, i.e. CPM2017 [29], Kumar [26], visual field from TCGA [30], and bone marrow [31]. The images of size 256 ×256 are extracted from more than 20,000 Whole Slide Images (WSI). It contains a total of around forty-seven thousand labeled nuclei. The nuclei are labeled as instances and five classes, i.e., neoplastic, non-neoplastic or epithelial, inflammatory, connective, and dead nuclei. The dataset is split into training, validation, and testing split. We follow the same splits as followed by earlier literature in order to get reproducible results. We use limited pre-processing on the data. We removed a few blank images present in the dataset. We extract the canny edges from the labeled instances to use them for training our network head which predicts the edges of the nuclei.

We developed the proposed network using Keras, an open-source python library for deep learning. The model is trained via Nvidia RTX 2060 graphical processing unit (GPU), accelerated by CUDA 11.0. We minimize the losses using the Adam optimizer with the mini-batch size of five. The learning rate is set to 0.004 at the start of the training. When the model stops learning for five consecutive epochs, the learning rate is decayed by a factor of 0.3. The training is considered as completed if the validation loss stop improving for fifteen successive epochs.

We utilize PQ for comparative analysis as discussed by Graham et al. [5]. We use binary PQ (bPQ), and multi-class PQ (mPQ) as suggested by Gamper et al. [9]. bPQ assumes that all the nuclei are associated with only a single class, while mPQ calculates PQ for an individual class of nuclei and then finds the average of the individual PQ. One of the reasons for using mPQ is that it is not affected by class imbalance. To know, specifically, how our method performs over each tissue individually, we calculate bPQ, and mPQ for each of the 19 issues in the dataset individually. We report tissue-wise results and the average across the tissues in Table 1. Table 1 also show the comparison of the suggested framework against top-performing methods i.e. HoVer-Net [5], DIST [6], Micro-Net [32], Mask-RCNN [20], and PointNu-Net [4].

The suggested framework achieves state-of-the-art performance compared to the top performers both for instance segmentation and classification (refer to Table 1). We report average PQ for each nuclei class in Table 2. The suggested framework has significantly improved tissue classification, especially the connective and dead cells.

We observe that the suggested framework produces improved semantic segmentation compared to the top performers. Since bottom-up approaches rely on semantic segmentation, we have compared the proposed framework specifically with bottom-up approaches. The comparison is made using dice metrics. The comparison is reported in Table 3. The table also verifies the fact that the semantic segmentation of the bottom-up approaches improves their instance segmentation and classification, as demonstrated by [4].

We present visual results, under variable conditions, of our semantic and instance segmentation in Fig. 5, while classification results in Fig. 6. We have also presented our visual results compared to the top performers in Fig. 7. The images in Fig. 7 also represent harsh conditions. We further compare our visual results with the top two performers, PointNu-Net [4] and HoVer-Net [5] (reported in Fig. 8). The suggested framework successfully segments and classifies partial or smaller nuclei. In comparison, the existing methods fail to segment smaller or partially present nuclei.

Although, the significance of the results of our framework can be observed in Table 1. However, we perform an in-depth analysis to analyze our framework performance in further detail. Based on the detailed results on individual tissue sites, we plot a box plot (Fig. 9). The left boxplot in the figure is drawn, for classification, while the boxplot on the right side is drawn for instance segmentation. The boxplot helps us to analyze various aspects of the framework. 1) The color dots represent outliers among the results. We can observe that the proposed framework produces the least number of outliers, demonstrating the framework’s generalization capability across multiple organs. 2) The color boxes show the interquartile range of the frameworks. The suggested framework results in a limited quartile range in comparison to the state-of-the-art, representing the framework’s stability.

The complexity analysis of the network is an important factor in histology since a single whole slide may be of the size of 100,000x100,000 pixels. Although it is an essential comparison, it is rarely mentioned in the literature. We have compared the complexity of the suggested model with the state-of-the-art models. The comparison is in terms of the training parameters. It represents the training and inference complexity of the models. The proposed three-head model has a total of 2.74 million parameters. However, if we use a separate feature extractor for improved nuclei classification, our total parameters are increased to 4.19 million. Compared to the state-of-the-art models, the parameters are far much less. The nearest competitors are DIST [6], and HoVer-Net [5]. The figurative comparison of the parameters are shown in Fig. 10. It is to be noted that the complexity of the post-processing used by the suggested framework and the state-of-the-art are not reported. It is due to the reason that the resources utilized by the post-processing are negligible compared to the models.

For an in-depth analysis of the proposed model, we performed ablation studies related to the model’s components. This study helps us to validate different aspects of the proposed method. We specifically perform four studies. First, we use a formal watershed algorithm instead of a controlled watershed. We observe that a formal watershed algorithm leads to poor instance segmentation. The use of controlled watersheds is encouraged in the previous literature compared to formal watershed methods; thus, this re-verify the previous literature. Second, we observe the framework without using pixel grouping. If we do not use pixel grouping, we get multiple class suggestions for each nuclei instance since one of our three head network heads produces pixel-wise classes. This problem is consistent with other bottom-up approaches as well. This makes the end-to-end networks for nuclei classification in histology less feasible. It also hints at the superiority of bottom-up approaches over the existing end-to-end state-of-the-art networks. Third, we use the same class weights for each nuclear category. This causes reduced performance based on the fact that the histology dataset has a high nuclei class imbalance. Because of the class imbalance, we see the worst performance of the existing methods for dead nuclei. Therefore we suggest the use of a weighted loss function for classification. Finally, we observe the framework using the same feature extractor for semantic segmentation, edge proposals, and pixel-wise classes. This has almost no effect on semantic or instance segmentation. However, using the same feature extractor reduced the classification performance. The use of a separate feature extractor is also suggested by previous works, such as Graham et al. [5]. However, using a separate feature extractor increases the number of parameters. The results of the ablation studies are summarized in Table 4.