Nucleus-aware Self-supervised Pretraining Using Unpaired Image-to-image Translation for Histopathology Images

Recognizing the glandular structures is beneficial for the diagnosis and grading of carcinomas originating in several organs, including the prostate, breast, and colon[41]. In these organs, nuclei are specially distributed due to the existence of glands. We simulate the phenomenon by positioning some of the nuclei around the lumens, which are represented by deformed, empty, and overlapping ovals. To be specified, we randomly determine the center point, orientation angle, and length of the axis for each oval. A polar coordinate system is then built upon the oval center, with nucleus centers being evenly distributed around the oval at each polar angle. The radial distance of each nucleus is perturbed, and more nuclei are randomly added at each polar angle.

We stylize the masks of nuclei and place them on the centers determined by nuclei distribution. Nuclei masks are generated as random polygons smoothed with Bézier interpolation, and we distinguish epithelial cells which form the glandular structures and other cells by placing their nuclei in different channels. The intensity values of nuclei are perturbed to add stochasticity, and the nuclei boundaries are drawn in a separate channel in order to guide the network to focus more on boundary recognition. Moreover, we allow some of the nuclei to slightly overlap with each other, which is a common phenomenon in histopathology. Examples of the prepared mask images are shown in Fig. 2.

We collect 1093 WSIs scanned from H&E stained colorectal tissues in the First Affiliated Hospital of Zhejiang University. Both normal and malignant slices are included and cropped into patches with a size of 512$\times$512 pixels. Finally, a total of 160,000 unlabeled histopathology images at 40$\times$ objective magnification (0.25 $\mu$m/pixel) are prepared for pretraining.

Kumar[44] is a public nuclear instance segmentation dataset containing 30 annotated tiles extracted from 30 patients in 18 institutes. These images are scanned and cropped from H&E stained tissues at 40$\times$ objective magnification (0.25 $\mu$m/pixel) from 7 different organs (prostate, breast, colon, liver, kidney, stomach, and bladder) with the size of 1000$\times$1000. According to the original work, 16 images are used for training and 14 for testing. We only use the dataset for the downstream task in the transfer learning protocol due to the extremely small amounts of data.

This dataset consists of colorectal histopathology images covering 9 classes for classification[45]. All images with the size of 224$\times$224 pixels at 20$\times$ objective magnification (0.5 $\mu$m/pixel) are obtained from the tissue bank of the National Center for Tumor diseases (NCT). We only utilize two of the classes, namely the colorectal adenocarcinoma epithelium (TUM) and normal colon mucosa (NORM), resulting in 23,082 images for training and 1,976 images for testing. Similar to [13], we randomly sample 20% of the training data for validation.

Lizard[46] is the largest nuclear instance segmentation and classification dataset currently available. It consists of 6 subsets (GlaS, CRAG, CoNSeP, DPath, PanNuke, and TCGA) of colorectal histopathology images with an average of 1,016$\times$917 pixels at 20$\times$ objective magnification ($\sim$ 0.5$\mu$m/pixel). Nearly half a million nuclei are boundary-labeled and classified into 6 subtypes (epithelial, neutrophil, lymphocyte, eosinophil, plasma, and connective tissue cells). The subset TCGA is currently not available and is excluded from our study, resulting in a total of 238 images. The two largest subsets (the Dpath subset with 69 images and the CRAG subset with 64 images) are used for downstream tasks. We equally divide the two subsets into 3 parts for training, validating, and testing.

Crag [47] is a public dataset proposed for segmenting colorectal adenocarcinoma glands. It is composed of 213 H&E images with a size of approximately 1,512$\times$1,516 pixels at 20$\times$ objective magnification ($\sim$ 0.5$\mu$m/pixel). We follow the official settings and divide the dataset into 173 images for training and 40 images for testing.

The PanNuke dataset [48] comprises 7,901 tiles with 256$\times$256 pixels at either 20$\times$ or 40$\times$ magnification, obtained from over 20,000 whole slide images (WSIs) of 19 organs. The dataset includes annotations of 189,744 nuclei belonging to 5 different cell types, namely neoplastic, non-neoplastic epithelial, inflammatory, connective, and dead cells. Following the official settings, the dataset is divided into a training set, a validation set, and a test set.

We also collect 260 images with the size of 10,000 $\times$ 10,000 pixels at 40$\times$ magnification from the same source with the In-house dataset. The dataset is composed of 130 malignant images and 130 normal images, which are annotated by an expert pathologist. A 65%-15%-20% split was used to split data for training, validation, and testing. The dataset is used for multiple instance learning. A bag indicates a set of patches with the size of 512$\times$512 extracted from an image Positive bags are malignant images that contain cancerous cells, and negative bags are images that only contain normal cells.

The pretraining quality is accessed using the transfer learning (TL) and semi-supervised learning (SSL) protocol. For TL, the framework is pretrained from scratch on the in-house dataset, followed by supervised training on various downstream tasks with end-to-end fine-tuning. For SSL, models are pretrained on all training data and fine-tuned on parts of the dataset.

(1) Classification. Accuracy (Acc) and F1 score are used for classification tasks, including patch-wise classification and multiple instance learning. (2) Nuclear Segmentation and Detection. Aggregated Jaccard Index (AJI)[44] is reported to evaluate the segmentation quality, and F1 score with an IoU threshold of 0.5 is used to evaluate the detection quality. (3) Multi-class Nuclear Segmentation and Detection. If the further classification of nuclei is required, the segmentation quality is evaluated with multi-class panoptic quality (mPQ+) proposed in the CoNIC challenge[17]. The metric is the average across per-class PQ, whose statics are calculated over all images. F1 score averaged across all classes is reported to quantify the detection quality. (4) Gland Segmentation. We report object-level Dice and Hausdorff distance, which is used in the GlaS challenge[49] to evaluate the instance-level gland segmentation quality. We run each downstream task five times with different random seeds and report the mean, standard deviation, and statistical analysis based on independent two-sample t-test.

ResNet-50 is used as the feature extractor in the bottom-up pathway of $S$. The normalization layers in the FPN and the segmentation branch are implemented with Group Normalization (GN) instead of Batch Normalization (BN) to alleviate the problem caused by the small batch size[50]. $D_{G}$ and $D_{S}$ maintain the design in StyleGAN2-ADA and CycleGAN, respectively. We rescale the input images to be 40$\times$ in order so that networks can concentrate on fine-grained details. Images are then resized in a range of 0.8$\times\sim$ 1.0$\times$ of original size and cropped to 256$\times$256 before feeding to our framework. We optimize the framework with batch size 12 and Adam optimizer with $\beta 1=0$, $\beta 2=0.99$, and $\epsilon=10^{{}-8}$. Hyperparameters for the instance segmentation branch are set following Mask-RCNN. We also follow the settings of StyleGAN2 when training $G$, except that the lazy regularization and the path length regularization are disabled. The weight-balancing hyperparameters $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are empirically set to 2.0, 10.0, and 2.0. Both the $\gamma_{g}$ and $\gamma_{s}$ for $R_{1}$ regularization are set to be 1.0. For the two-stage training strategy, we train the framework for 40k iterations with learning rate of $4.0\times 10^{{}-4}$ in the first stage, and 25k iterations with learning rate of $1.0\times 10^{{}-4}$ in the second stage. The whole pretraining process is conducted on 4 GeForce RTX 2080 Ti GPUs.

For classification tasks on Kather, we initialize the ResNet backbone with our pretrained model and build a classifier head with adaptive average pooling, fully-connected layer, and softmax on top of the backbone. For the nuclear detection and segmentation task on Kumar, we follow [15] and use Panoptic FPN[55] as the segmenter. The weight of FPN is initialized by our pretraining approach, whereas the semantic segmentation branch and instance segmentation branch are initialized randomly. For multi-class nuclear segmentation and detection on PanNuke and the subsets of Lizard, we utilize Mask-RCNN, whose FPN is initialized by our method. For gland segmentation on Crag, we take UperNet[59] as the semantic segmenter and initialize the ResNet-50 backbone. In this task, both the objects and contours are predicted to help separate the touching instances following [57]. Multiple instance learning experiment is implemented following C2C [60], whose backbone is replaced with ResNet-50. The learning rate, batch size, and optimizer of each experiment are determined with grid search.

As shown in Table 2, although no extra annotation is used in our pretraining framework, we outperform supervised pretraining on ImageNet (p$<$0.05) and Mormont et al. (p$<$0.05) on Kumar. Moreover, our method produces better results than previous SOTA pretraining approaches (p$<$0.05, compared with MoCo v2), indicating that our nucleus-aware method is more effective for the dense-prediction task and generalizes better to histopathology images from various organs. The per-organ results reported in Table 4 indicate that our method consistently benefits nuclear segmentation across various organs. We not only improve the performance on the organs with glandular structures (e.g., colon, prostate, and breast), but also benefit that without the special structure (e.g., bladder). We also compare with the SOTA methods for nuclear segmentation on Kumar. To this end, we use HoVer-Net[4] as the segmenter, and replace the Preact-ResNet-50 backbone with the pretrained ResNet-50, similar to the previous work [61]. As reported in Table 3, it is interesting to find that all the pretraining methods improve the performance of Hover-Net (Dice), among which we achieve new SOTA results on Kumar.

Cancer classification is performed on the Kather dataset. We only fine-tune on 1% training data to make the task more challenging and exacerbate the differences amongst pretraining approaches. Results in Table 2 indicate that ImageNet provides more effective initialization than most of the pretraining methods. The results are in line with the finding that the self-supervised methods do not always achieve superior results against ImageNet [52]. Despite this, our method not only significantly outperforms other self-supervised methods with an increase of at least 1.12% for accuracy and 1.08% for F1 score (0.9590 vs. 0.9478 and 0.9564 vs. 0.9456, compared with MoCo v2, p$<$0.05), but also achieves slightly better results than ImageNet. One possible explanation for these results could be that the cancer classification task is highly dependent on the properties of nuclei, which our pretraining method can help recognize better.

We perform gland segmentation tasks on the Crag dataset. It is noteworthy that although our pretraining method is nucleus-aware, we also provide stronger initialization for gland segmentation than supervised counterparts (ImageNet, p$<$0.05 and Mormont, p$<$0.05, Obj Dice). We also achieve superior results than other self-supervised methods, although the improvement is not significant (0.8649 vs. 0.8616, p=0.28, compared with Ciga). The improvements can be explained by our special design of the nuclear distribution in our mask images, which is similar to that of epithelial cells around the glands.

On the Dpath subset of Lizard, the proposed method outperforms other pretraining methods for mPQ+. However, no improvement in F1 score is observed compared to MoCo v2. We attribute the results to the trade-off between nuclear detection and subtyping, which is more obvious in the proposed approach because we do not classify the instances in our segmentation branch during pretraining. When experimenting on the CRAG subset of Lizard, we achieve the best results in terms of both mPQ+ and F1.

We also experiment on a more diverse dataset to further evaluate the transferability of our pretrained model in case multi-organ and multi-class nuclei need to be segmented. The experiments performed on PanNuke indicate that our method outperforms DiRA (p$<$0.05), which is the previous SOTA pretraining method. We can learn from the per-organ results in Table 5 that both the epithelial-like structures (e.g., those in breast, colon, and liver) and non-epithelial-like structures (e.g., those in bladder) benefit from our pretraining methods. The results suggest the robustness of our pretrained model that benefits various downstream situations.

The multiple instance learning experiment is performed on the In-house-MIL dataset to evaluate whether our method benefits the special two-stage model for MIL. The original work [60] used ImageNet-pretrained backbone for sampling and aggregation. We also find the strategy effective compared with initializing the backbone randomly, which can hardly provide discriminative features after sampling. Moreover, it is observed that the generalized self-supervised pretraining methods fail to provide stronger initialization, whereas the pertaining methods for medical images benefit more compared with ImageNet. In such a special task, our method still demonstrates its superiority compared with ImageNet (p$<$0.05), and outperforms other pretraining approaches.

We use the CRAG subset and the Dpath subset of Lizard for the nuclear object detection and segmentation experiment. We fine-tune Mask-RCNN with different annotation budgets (10%, 25%, and 50%) and present the results for each subset in Table 6. Our pretraining approach improves the performance on both subsets and all of the metrics over prior methods significantly (p$<$0.05). On the CRAG subset, it is striking to find that we just need 5% of labeled data to match the best performance obtained from other methods that need 25% annotation budgets (0.2478 vs. 0.2522, mPQ+). Our method also performs well on the Dpath subset, where MoCo v2, Simsiam, and DiRA fail to provide effective pretrained models in most cases, whose results are even worse than random initialization. We attribute the decreased performance of these approaches to insufficient pretraining data. Specifically, these approaches may be overfitted to the pretext tasks in the Lizard training dataset, which contains only 194 images for pretraining. The proposed pretraining method, on the other hand, does not have this issue and yields more robust results.

We pretrain on the whole training set and validation set of Kather without any annotations, and fine-tune the classifier with different annotation budgets (0.2%, 0.5%, and 1%). Similar to [13], all of the data regimes are extremely low due to the high baseline performance (0.9532 for Acc with 2% training data). Table 6 demonstrates that we outperform the baseline whose weights are initialized randomly (p$<$0.05). It is noteworthy that our pretraining method focuses on fine-grained details where most of the nuclei are recognized. This is confirmed in Fig. 5 with Grad-Cam [62] visualization algorithm which generates a heatmap that highlights supportive pixels in the histopathology image for the classifier. It is also interesting to find that we not only focus on nucleus-level features, but also capture cell/tissue-level features, which may interpret the superior performance on gland segmentation in 4.4.3. Such multi-level features are beneficial for classification, where we outperform other discriminative self-supervised learning approaches (i.e., MoCo, Simsiam, Ciga) significantly (p$<$0.05). However, the performance boost is not as significant as segmentation tasks compared with DiRA, which captures stronger global representation by incorporating discriminative self-supervised learning with generative and adversarial ones.

We examine the effect of the quality of synthesized mask images. As illustrated in the corresponding section in Table 7, extra information brought by the nuclei distribution and stylization is introduced. We begin with a random distribution and binary nuclei mask design following [14], then progressively add the glandular structure and stylize the nuclei. Generally, we can observe the performance gain for both the transfer learning and semi-supervised learning with the introduction of distribution priors (p$<$0.05). Stylization benefits most cases, but slightly degrades the segmentation performance on Kumar.

To analyze the effect of the network design, we investigate different architectures of our framework. As summarized in Table 7, we divide the design into three main parts: The co-modulation design for pathology generator $G$, the adaptive discriminator augmentation strategy (ADA) for $D_{G}$, and the segmentation guided strategy. For the first part, we find it effective in our transfer learning protocols. However, in our semi-supervised learning protocols where the data for pretraining is limited, the complex generator design is not beneficial due to the over-fitting problem. We alleviate the problem by augmenting the pretraining data with ADA, which effectively boosts the performance for semi-supervised learning, while also further improving the performance for transfer learning. Moreover, we discuss the benefits of instance segmentation guided strategy (ISG). The results in Table 7 and Table 8 show that the semantic segmentation guided strategy fails to offer useful information, supporting our analysis in 3.4. On the contrary, the instance segmentation guided strategy effectively improves the performance, especially for transfer learning experiments where the improvement of 1.68% is observed on Kumar and the improvement of 1.31 % is observed on Kather (both in terms of F1 score, p$<$0.05).

Due to the inconsistent convergence rates of $G$ and $S$, we design a two-stage pretraining strategy. Table 7 and Table 9 investigate the effect of the extra training stages. Overall, the additional training stage yields superior performance than pretraining for a single stage (p$<$0.05 for transfer learning). We observe no significant performance improvement with more training stages.

We evaluate the effect of initializing the ResNet backbone, FPN, and all the structures we can initialize (including the FPN and the instance segmentation branch) in Table 10. Initializing FPN demonstrates better performance than merely initializing the encoder backbone. It is also inspiring to find that the performance can be further improved by further initializing the instance segmentation head on Kumar. However, when additional instance segmentation heads are further initialized, we observe a minor improvement on Kather but degraded performance on Lizard. This may be attributed to the different roles of the segmentation branch during pretraining and fine-tuning. In downstream tasks, when the segmentation branch further classifies the nuclei, initializing the heads of the segmenter might cause the network to ignore the differences among nuclei. Therefore, we only initialize the FPN for all the dense-prediction tasks.