MSA-MIL: A deep residual multiple instance learning model based on multi-scale annotation for classification and visualization of glomerular spikes

Recently, deep learning technology as a class of multi-layer neural network learning algorithms can automatically learn hierarchical feature representations of the input data and form higher abstract features (attribute categories or features) by combining low-level features through deep nonlinear network structures. At present, deep learning methods have been widely used in the auxiliary diagnosis of diseases, and have made great progress in handling CT images [15, 16, 17], MR images [18, 19, 20], retinal fundus images [21] and skin images [22]. Among the available deep learning methods, deep convolution neural network (DCNN) and convolution residual network (CRN) are the most popular algorithm.

In the field of pathological images classification, especially for the breast cancer pathology classification, deep learning has shown good performance in dealing with the breast cancer data. Spanhol et al. [23] used the AlexNet network to classify breast cancer, and the recognition rate obtained was 6$\%$ higher than that of traditional algorithms, but the accuracy remains low and could not conform to the clinical requirements. Wei et al. [24] used GoogleNet as the basic network architecture and used the labels of classes and subclasses as a priori knowledge to construct the breast image CNN (BICNN) model. Although this model arrives very high accuracy of classification, network deepening lead to the degraded efficiency. The ResNet [25] proposed in $2015$ is currently the most widely used CNN feature extraction network, which can be used for classification. It avoids the problem of network degradation with the number of network layers increasing. The ResNet introduces the residual network structure, which can deepen the network depth and achieves better classification effect. However, due to the specificity of renal pathological lesions, the classification methods that directly extract the features of the entire image cannot learn the fine-grained lesion features. Therefore, the classification results are not acceptable and can not apply to clinical assistant diagnosis in the pathological image field. Additionally, the current classification methods are lack of interpretability, and the classification results of the model are invisible. Thus these classification models cannot convince the clinical doctors. For the segmentation of pathological images, the U-Net proposed by Ronneberger et al. [26] is the most widespreadly used segmentation framework based on CNN backbone and is one of the most influential models for medical image segmentation because of its small parameters and good segmentation effect. The U-Net combines the same number of up-sampling and down-sampling layers, which can generate a segmented image from the entire image by the forward algorithm in a fashion of end to end. At the meantime, the skip connections across the upper and lower sampling layers ensure the gradual preservation of information. When this network was first proposed, it was utilized for cell wall segmentation. Later, it has excellent performance in the detection of lung nodules [27] and the extraction of blood vessels on the fundus retina [28].

Multiple instance learning [29] (MIL) is a weakly supervised learning method, which builds a classifier for multiple instance bags with classification labels and predicts unknown multiple instance bags and can realize the visualization of classification results without increasing the labeling cost. In MIL, the training set consists of a set of bags with classification labels, and there are several instances without classification labels in each bag. If the bag contains at least one positive instance, the bag is marked as a positive bag. If all instances of a bag are negative instances, the bag is marked as a negative bag. There are some researches in the literature which regard the problem of image classification as the problem of multiple instance learning [30, 31, 32]. In recent years, MIL has been successfully used in pathological image classification [33]. Yan et al. employed the MIL framework to complete the classification of colon cancer image by regarding the pathological image as a bag and dividing the image into multiple patches as an instance [34]. Hou et al. [35] Proposed a multiple instance convolution algorithm based on EM algorithm to classify gliomas. Campanella et al. [36] proposed a deep learning system based on multiple instance learning, which only uses the diagnosis report as the label of training, and classifies the pathological images of prostate cancer, basal cell cancer and breast cancer transferred to axillary lymph nodes. However, the label of the positive bag is not the real label of instance, the stability of the label is greatly challenged, which leads to incorrect classification results to a certain extent.

In this work, we propose a multi-scale annotation (MSA) method which is a data processing method used for enhancing the learn capacity of MIL. It overcomes the defect of unstable labels by adding instance-level labels. Specially, the MSA hires image-level annotation and location-level annotation (box-level annotation) to annotate images, respectively. For the image-level annotation, the pathologist gives a label of the whole images when cropping the image into tiles and each label of the tile is deemed as the label of whole image. Suppose $x$ denotes a tile, $y$ represents the label of the tile, $i$ is bag index, $j$ is tile index. For the negative image, we obtain weakly negative labeled data set $W_{n}(x_{w}^{(ij)},y_{w}^{(ij)})$. For positive image, we get weakly positive labeled data set $W_{p}(x_{w}^{(ij)},y_{w}^{(ij)})$. While for the location-level annotation, the pathologist annotates the true location of glomerular basement membrane spikes with a box in the glomerulus and gets the reinforced data set $S(x_{s}^{(ij)},y_{s}^{(ij)})$. Due to the location-level annotation only labels the location of spikes with a box, it is a reinforced data set of positives. Therefore, the MSA method adds a small number of instance-level real labeled data into training data set, strengthening the supervision information in the training process of instance-level classification networks. In Fig.2, we give an illustration of the MSA method display the MSA method involving the image-level annotation and the location-level annotation.

The weakly labeled training set $W_{n}(x_{w}^{(ij)},y_{w}^{(ij)})$, $W_{p}(x_{w}^{(ij)},y_{w}^{(ij)})$ and the reinforced data set $S(x_{s}^{(ij)},y_{s}^{(ij)})$ are gathered and formed the total training data set $N(x_{N}^{(ij)},y_{N}^{(ij)})$, It would be used to train the instance-level classification networks. The specific training method will be described in detail at the next section.

To solve the problems of instability of labels in MIL and non-visualization of classification results in the traditional classification methods, a MIL method based on multi-scale annotation is developed in this section to achieve the classification of glomerular spikes and visualization of the classification results. The MSA-MIL model mainly contains three phases. In the first phase, The U-Net is used to delineate the boundaries of the glomeruli, and some other digital image processing methods such as cavity filling and retention of connected domains are used to post-process the segmentation regions of the glomeruli. For the second phase, the obtained data by the U-Net is fed into the ResNet model under the MIL framework for training and a classifier is obtained. In the process of network training, we add a small amount of reinforced data via the MSA method for strengthening the learning capability of the network, so as to obtain instance level feature representation with rich semantic information. In the last phase, the predicted scores of each tile in the glomerular images are aggregated to get the classification results and visualization of spikes. The network architecture is shown in the Fig.3.

Segmentation of Glomerulus: In the classification of spikes glomeruli, Since the major regions of concern is the glomerulus itself, the outside area of the glomerulus in the image can be precluded. The significant advantage of preprocessing the image is that the succeed learning algorithm mainly focuses on the important area making the algorithm, learns the characteristics of the glomerular internal lesions more effectively, and improves the accuracy of the classification algorithm. However, due to the internal structure of the glomeruli is complex, it is ineffective for traditional methods to segment the glomeruli. In this paper, we take U-Net as a data preprocessing step to segment glomerular edges because of its excellent segmentation performance at dealing with medical images. In order to ensure the stability of model training and optimize the effect of model training, we unify the size of glomerular image in the same range. Firstly, the glomerulus was randomly clipped from the WSIs magnified by $40\times$, and then the size of the glomerulus image was uniform $1100\times 1100$, and the pixel was $0.25\upmu$m/pixel. The pathologist marks the outer edge of the glomeruli in the image. The original image and the masked image are put into the U-Net for training at the same time, in which the cross-entropy loss function is used.

Instance-level classification with multi-scale annotation: We resize the resolution of the segmented glomerular images as $1100\times 1100$ for keeping the size of each image identical. We take images with spikes as positive bags and images without spikes as negative bags. For the images of positive and negative bag, we divide the extracted regions of them by U-Net into many tiles in which each tile is an instance in the bag, and the label of glomerular image is the label of tile in this image. According to the above analysis, the positive bag will contain at least one positive tile(with spikes), while the negative bag only contains the negative tile (without spikes). To ensure the features of spikes integral, we employ overlapping method in tile partition, and the overlapping step size is half of tile size. The selection of tile size is related to the size of specific lesions and would be discussed in experimental part.

By using the method of transfer learning, the classification training is first carried out on the open pathological data set, and the trained ResNet parameters and classifier $f$ are taken as the initial parameters and classifier of MSA-MIL training. In the training process, we use the image-level labeled tiles as the weakly labeled data set $W(x_{w}^{(ij)},y_{w}^{(ij)})$, and the location-level labeled tiles as the reinforced data set $S(x_{s}^{(ij)},y_{s}^{(ij)})$, where $x$ represents a tile, $y$ denotes the label of tile, $i$ is bag index, $j$ is tile index. The reinforced data $S$ set is stored in a temporary training set $D(x_{d}^{(ij,k)},y_{d}^{(ij,k)})$, where $k$ is the iterative index. For each bag, we first utilize initial classifier $f$ to class the data set $W$ and obtain the classification result of each tile in the bag, denoted as $Q_{i}=q_{(ij)}$. Then the $Q_{i}$ is processed by classifier $f$ and a probability vector $P_{i}=p_{(ij)}$ is obtained. Therefore, the tile with the highest positive probability can be expressed as $j=argmaxp_{(ij)}$. Take the positive instance with high probability in each positive bag, take the negative instance with high probability in each negative bag, and append them in the temporary training set $D(x_{d}^{(ij,k)},y_{d}^{(ij,k)})$. Finally, Using temporary training set $D$ to train the classification network and obtain a updated classifier $f$. By repeating the above three steps to iteratively fine-tune the network parameters, we get the final instance-level classification network for spikes.

In the process of updating classifier $f$, we use the cross-entropy loss by comparing the output $\hat{y}_{i}=f(q_{(ij)})$ with the target $y_{i}$ for each bag as follow:

and the final loss is the weighted average of small batch loss.

Actually, the training of multiple instance learning in the network is to update the training set so as to finely tune the network. In multiple instance learning, there must be at least one positive instance classified by some classifiers. Instead, for a negative bag, all instances must be classified as negative instances. Given a bag, all the instances will be classified and sorted in detail according to its probability. If the bag is positive, the top sample positive probability should be close to 1; if the bag is negative, its positive probability should be close to 0.

Classification and Visualization of spikes. We use the sliding window aggregation method to classify spikes. By sliding a fixed size of window from top left to bottom right of glomerular image, we can obtain the classification probability for each window. The window can be regarded as a positive instance if the probability exceeds 0.5, and the window is classified as negative instance when the probability is less than $0.5$. For an input image, it is classified as positive category if it contains a window of positive instance, the image is considered as negative category when all windows are classified as negative instance.

As for images are classified as positive category (with spikes), we need further to give the specific location of tile with high probability of lesions such that the classified result is interpretable. After computing the classification probability of each window, we connect the probability of all of window and form the probability distribution matrix of the whole image of glomerulus. According to the probability distribution matrix of the spikes, we can obtain a thermogram of spikes. The final heatmap of glomerular spike can be drawn by superposing thermogram the with a certain transparency on the original image.

The data of renal biopsy optical microscope slides are collected from the Second Hospital of Shanxi Medical University between $2014$ and $2019$ and the Shanxi Provincial People’s Hospital between $2017$ and $2019$. The slides are stained with periodic acid-silver metheramine (PASM) and the slice is digitized as a whole slide image (WSI) by KF-PRO-005-EX digital slice scanner (resolution $0.25\upmu$m/pixel) with $40\times$ objective and $10\times$ eyepiece. All of the images are obtained by clipping the image with magnification of $400\times$.

Training dataset for segmentation. Two pathologists first annotate and record the positions and coordinates of the glomerular in WSI by utilizing an image annotation system. Then, the pathologist cut the glomerulus as patch with the size of $1100\times 1100$ pixels. Finally, The pathologist marked the edges of the glomerulus patch at the accuracy of pixel-level, and the obtained mask is served as the label of glomerulus segmentation. We apply Dice coefficient (DC) to determine that the labeled data is effective. If the DC value between two labeled data obtained from two pathologists is greater than $0.8$, then the two labeled data is effective and we choose one of them as the training data. Repeating the above operating procedure, $1015$ images of glomerular were labeled at pixel-level in which $914$ of them is choose as the training set and the remained $101$ images is used as the test set.

Training dataset for classification. Two pathologists annotate the glomeruli patch at the image level. The annotated results have three classes of positive (denoted as $+$), negative (denoted as $-$), unable to determine (denoted as $+,-$). In order to ensure the reliability of the data, only the glomerular patch diagnosed as the same category (negative or positive) by two pathologists is added in the training data set. Additionally, to obtain reinforced training data, the glomerular image of positive category is annotated as a typical spikes by a box with $50\times 50$ pixels at location-level. Finally, there are $1267$ images with $653$ positive glomeruli and $614$ negative glomeruli in the data set, and $250$ location-level labeled images are produced from $653$ positive glomeruli. We divide the dataset and select $252$ images as the test data set to evaluate the performance of the proposed model, and the remaining $1015$ images is as the training data set.

To quantitatively evaluate the performance of the proposed model for classifying spiked glomeruli, we employ four metrics such as the Precision, the Recall, $F_{1}$-measure and the Accuracy in aspects of verifying the efficiency and accuracy of the proposed model. The mathematical formulas is listed as follow:

where $TP$ (true positive) denotes that the glomerulus with spike is correctly classified; $FP$ (false positive) represents that the glomerulus without spike is wrongly classified as the glomerulus with spike; $FN$ (false negative) indicates that the glomerulus with spike is wrongly classified as the glomerulus without spike; $TN$ (true negative) means that the glomerulus without spike is correctly classified.

Besides, we utilize the Receiver operating characteristic (ROC) curve to measure the sensitivity and specificity of the proposed model. For the coordinate of ROC curve, the horizontal and vertical axis represent false positive rate ($FPR$) and true positive rate ($TPR$), respectively. The specific formulations is given as follow:

We can from the ROC curve calculate the AUC value, which is the area under the ROC curve and is a standard measuring the quality of the classification model. Generally, the value of AUC is between $0.5$ and $1.0$. The larger the AUC value, the better the model performance.

After dividing the data set as training data and test data, we use the training set containing $1015$ glomerular images to train the MSA-MIL. Fig.4 displays the change of the loss and accuracy with training epoch increasing. we can see that the training of the model is effective, and the model converges at the $10$th epoch with lower loss and higher accuracy.

We use a sliding window aggregation method to visualize the spikes of glomeruli. We choose the same window size as the tile size to classify the images in turn, and obtain the classification probability of each window. By connecting the probabilities of all tiles in an image, a probability distribution matrix can be formed. According to the probability distribution matrix, we can draw a heatmap which is superimposed on the original image with a certain transparency to obtain the visualization results of glomerular spikes. The heatmap can reflect the location of the spike in the glomerulus at a certain extent. It solves the unexplainable problem of the current classification network. At the same time, the doctor can understand the basis of the algorithm classification through the heatmap of the spike, so as to trust the result of the classification.

Fig.5 shows the visualized results of four glomerulus images by the proposed model. The first column displays four original images of the glomeruli, the second column shows the ground truth for training U-Net. the third column is corresponding region extracting results of glomerular through the U-Net network and the fourth column gives the visualization results of spikes in the corresponding original images. The color from blue to red in the regions of heatmap indicates that the probability being a spike. The blue area indicates that the area has a low probability of being spikes, and the red area indicates that the area has a high probability of being spikes. It can be seen from the last column of Fig.5 that the MSA-MIL has achieved the goal of visualizing the location of the spikes in the glomeruli.

In order to obtain the best performance of the trained model, we need to choose the appropriate size of tile in dividing the image into many tiles and the ratio of positive and negative instances when updating the temporary training data set $D(x_{d}^{(ij,k)},y_{d}^{(ij,k)})$.

The size of tile has a great influence on the accuracy of model learning. Larger tile size can not learn small features, while smaller tile size will cause the model to fail to learn the integrable features. In order to select the appropriate tile, we use different sizes of tile (25, 50, 75 and 100) to test the proposed model and calculate value of the four metrics (Precision, Recall, $F_{1}$-measure and Accuracy) which are shown in Table1. The results show that the model achieve the best classification accuracy when the size of tile is set as $50\times 50$. Fig.6 shows different sizes of tile used in the proposed model, from which we can observe that the tile with the size of $50$ just fits into the basement membrane with spikes, fine-grained features cannot be learned with the tile size exceeding $50$ and the features are too fine and lack of integrity with the tile size under $50$. The result can be confirmed by the AUC value shown in Fig.7(a) in which the AUC value is the highest for setting the tile size as $50$.

When training the instance-level network, the ratio of positive and negative instances updating the training set is a parameter that needs to be adjusted. According to the setting of MIL, MIL uses the label of the image as the label of tile and thus the tile in the positive bag may be mislabeled. When updating the temporary training data set $D(x_{d}^{(ij,k)},y_{d}^{(ij,k)})$, the positive instances with the highest probability in the positive bag are put into the training set at each iteration. At this point, the number of negative instances will affect the accuracy of the model. we test two ratios of positive instance and negative instance with $1:2$ and $1:5$ and calculate the same four metric. The results in Table 2 show that the sufficient number of negative samples ensures that the model can learn the features of more negative instances, thus ensuring the accuracy of the classification results. The ROC curve shown in Figure.7(b) also confirm that the AUC value arrive the highest by setting the ratio of positive and negative instances as $1:5$, that is, the positive instance with the highest probability is selected from the positive bag each time, and the negative instances with the top five probabilities from the negative bag are selected and put into the training set for training.

In order to verify the effectiveness of MSA-MIL in classification, we selected several models for comparison. They are the MIL model without multi-scale annotation, the model without U-Net structure and the ordinary ResNet-18 classification model. For the proposed model MSA-MIL, U-Net is a necessary part of the network structure to remove extraneous regions outside the glomeruli. The features learned by the algorithm are mainly concentrated on the glomeruli, which improves the efficiency of the model. In addition, a common ResNet-18 is trained to classify spiked glomeruli and non-spiked glomeruli. The value of four metric obtained from the competing model are listed in Table 3. The results in Table 3 show that the multi-scale annotation training method proposed in this paper achieve higher accuracy. It indicates that the method of multi-scale labeling solves the problem of MIL label instability, and greatly improves the accuracy of classification. Furthermore, Fig.8 shows the ROC curves of the MSA-MIL, the model without multi-scale annotation and the model without U-Net. It can be seen from the ROC curve that the AUC value of the MSA-MIL is high and reaches $0.9889$. Therefore, the proposed model has better stability.

The proposed model is compared with pathologists in the classification performance of glomerulus and speed of marking the location of the spike. $252$ glomerular images in the test dataset were classified and labeled by a junior pathologist with three years of work experience and a senior pathologist with five years of work experience. They classify all the glomeruli and mark the location of the spike and record the time spent on processing each glomerulus. Table 4 shows the results of the average speed and performance of each glomerulus by the proposed model and two pathologists. As can be seen, the performance and speed of the pathologists are low and unstable because of subjective factors such as lack of clinical experience and work fatigue. Pathologists are prone to miss detection when classifying because the characteristics of the spikes are small and not obvious. MAS-MIL effectively helps medical pathologists to diagnose spikes and improves the efficiency of diagnosis.