Proportion Estimation by Masked Learning from Label Proportion

The proportional information of cancer subtypes is recorded as diagnostic information with pathological images in many diagnoses, such as programmed cell death ligand-1 (PD-L1) diagnosis [8, 14], chemotherapy, and lung cancer diagnosis [13, 15]. For example, the PD-L1 test is conducted to check if cancer immunotherapy will be helpful for a patient. Fig. 1 shows an example image (called core image) captured by a whole slide scanner. In the core image, over 10 thousand cells belong to three classes; positive cancer, negative cancer, and non-tumor cells. The PD-L1 rate is calculated by the number of positive tumor cells over all of the tumor cells in a tissue. Note that it does not include non-tumor cells. Counting all cells in the core image is almost impossible; thus, pathologists roughly estimate the PD-L1 rate without counting in clinical. Therefore, there is a demand to develop an automatic proportion estimation method.

A simple solution is to detect positive and negative tumor cells by a deep detection model [10] and calculate the proportion from the detection results. However, this approach does not use the useful information, PD-L1 rate, which has already been recorded as diagnosis information and requires a certain amount of cell-level annotation for positive and negative tumor cells. A small amount of annotation can be collected, but the performance of a network trained using insufficient training data is worse. Large datasets with sufficient variability must be collected to address inter-tumor heterogeneity and different staining characteristics, but it is time-consuming and labor-intensive.

Another solution is to estimate the proportion by regression or classification directly. The approach can train a network using proportion information without additional annotation since PD-L1 rates are recorded in clinical settings. However, these methods have two drawbacks. A PD-L1 rate is a ‘partial’ proportion of all cells, i.e., it is calculated only from tumor cells but ignores non-tumor cells. This ambiguity caused by without using the non-tumor region information makes training difficult to estimate proportion. Another disadvantage is poor interpretability for estimation. When pathologists use the estimated results in clinical, they check the cell distributions of three classes to know how the rate is calculated. A class activation map (CAM)-based approach [16], which visualizes contributed pixels to the output, is unsuitable for this task because class distributions of all tumor regions are required to estimate the proportion.

This paper proposes a proportion estimation method to estimate the proportion of PD-L1 tumor cells and output the distributions of three classes in a core image while keeping annotation costs as low as possible. As discussed above, estimating the proportion without tumor region information is difficult. Thus, we made a small amount of training data to train a cell detection network that detects tumor or non-tumor cells. This enables us to make a tumor region mask, which is used for estimating the proportion (PD-L1 rate). We proposed a masked ‘Learning from Label Proportions (LLP)’ that estimates the proportion by using the tumor cell region mask predicted by the cell detection network. In addition, we propose a weighted focal proportion loss to address data imbalance problems, where data imbalance often occurs in medical image applications. Experiments using clinical data demonstrate the effectiveness of our method. Our method achieved the best performance in comparison.

Our method aims to train a network that estimates the proportion $r$ of PD-L1 tumor cells in core images $I$. We use three types of annotation for the training. One is a cell position label, the second is a tumor cell region, and the third is the proportion label. For proportion labels, a proportion is often recorded as interval sections in clinical since pathologists roughly estimate the PD-L1 rate subjective without counting. For example, PD-L1 rates are either ‘0 to 0.01’, ‘0.01 to 0.25’, ‘0.25 to 0.5’, ‘0.5 to 0.75’, or ‘0.75 to 1.00’.

Fig. 2 shows an overview of the proposed method. Our method was designed as a two-stage structure to effectively use a small amount of cell-level annotation and a large amount of tissue-level annotation (proportion). We first detect tumor cells in core image $I$ to produce tumor mask $M$. Then, we estimate the proportion of PD-L1-positive cells of an input image $I$ using mask $M$.

We first train the cell detection network $h$ using the training data of cell positions, where $h$ produces the cell position heatmap for input image $I$ [9]. To train $h$, the ground truth of the heatmap $H$ is generated using the given training data of cell positions. The network $h$ is trained by minimizing the MSE Loss $L_{d}=\|H-\hat{H}\|^{2}$, where $\hat{H}=h(I)$ is an estimated heatmap by $h$. The peak points in $\hat{H}$ are detected as the cell positions in a testing phase, denoted as $\{\hat{{\bf p}}_{i}\}_{i=1}^{N_{d}}$, where $N_{d}$ is the number of detected cells in $I$. Note that $\hat{H}$ only contains the cell position information but not the tumor class information.

For each detected cell position $\hat{{\bf p}}_{i}$, the tumor classification network $g$ (consists of $g_{f}$ and $g_{c}$) estimates its class $\hat{y}_{i}$. The feature extractor $g_{f}$ extracts a feature map $F_{t}$ from $I$, and then the fully connected (Fc) layer $g_{c}$ estimates the tumor class $\hat{y}_{i}\in[0,1]$ for each cell position by inputting the feature vector $F_{t}(\hat{{\bf p}}_{i})$ at pixel $\hat{{\bf p}}_{i}$, where $\hat{y}_{i}>0.5$ indicates a tumor cell, otherwise, a non-tumor cell. To train this network $g$, we use the binary cross-entropy loss between the predicted score $\hat{y}_{i}$ and the ground truth $y_{i}$, where the loss is calculated at only the detected cell positions $\{\hat{{\bf p}}_{i}\}_{i=1}^{N_{d}}$. The detection results are denoted as $\hat{\mathcal{P}}=\{\hat{{\bf p}_{i}},\hat{y}_{i}\}_{i=1}^{N_{d}}$.

We generate a tumor cell mask $M$ using detection results $\hat{\mathcal{P}}$. In $M$, pixels around the tumor cell positions $\hat{\mathcal{P}}_{c}=\{\hat{p}_{i}|\hat{y}_{i}>0.5\}$ takes 1, otherwise 0, where the distance from the detected position to a positive pixel is less than $\alpha$. This mask is used to estimate the proportions.

As discussed in the introduction, the PD-L1 rate is given as a proportion interval; pathologists give either ‘0 to 0.01’, ‘0.01 to 0.25’, ‘0.25 to 0.5’, ‘0.5 to 0.75’, or ‘0.75 to 1.00’ for a core image. In previous works, the loss for proportion interval [2] outperforms cross-entropy. This loss mitigates the problem of overfitting by sacrificing strictness but at the expense of discriminability. In our problem setting, there are different intervals; the interval in ‘0 to 0.01’ is much smaller than that of others, and there is data imbalance, e.g., the number of core images belonging to ‘0.5 to 0.75’ is much fewer than that of ‘0 to 0.01’. These issues make training difficult.

We thus propose a weighted focal proportion loss that can mitigate these issues. This loss is designed inspired by the focal loss [7], which has been widely used for imbalanced data classification. The focal loss is a dynamically scaled cross-entropy loss, where the scaling factor decays to zero as confidence in the correct class increases [7]. We introduce this idea into the proportion loss [1], widely used in LLP. Let us denote $r_{k},(k=1,2)$ are the proportion of the positive and negative cells. The weighted focal proportion loss is defined as:

where $r$ is the ground truth of the proportion (PD-L1 rate), which takes the mean of the interval; e.g., $r=0.375$ for ‘0.25 to 0.5’. $\hat{r}$ is the estimated proportion, and $\gamma$ is a hyper-parameter. $\sum_{k}r_{k}\log\frac{r_{k}}{\hat{r_{k}}}$ indicates the KL-divergence between the truth and estimated proportion (proportion loss), $r_{1}=r$ indicates the proportion of the positive cells, and $r_{2}=1-r_{k}$ indicates that of the negative one. $|r-\hat{r}|^{\gamma}$ indicates that the loss decreases when the estimation becomes the correct value.

Figure 3 (Left) shows the plot of the focal proportion loss for each proportion interval when $\gamma=2$. In this graph, the loss for ‘0 to 0.01’ (blue line) has low gradients, and the loss around 3 % is lower than that of the neighboring interval ‘0.01 to 0.25’. It makes training difficult to distinguish the proportions from the neighbor intervals. This is because the interval of ‘0 to 0.01’ is much smaller than the others. Therefore, we add the weight for the hyper-parameter $\gamma$, where smaller $\gamma$ gives more significant gradients: we use $\gamma=0$ for ‘0 to 0.01’ and $\gamma=2$ for other intervals. Figure 3 (Left) shows the weighted focal proportion loss curves. The blue curve (‘0 to 0.01’) has larger gradients, and thus this makes training easy to identify the proportion in either ‘0 to 0.01’ or ‘0.25 to 0.5’.

For proportion estimation, we used Resnet-18 [5] pre-trained on Imagenet. The Adam optimizer [6] is adopted with a learning rate of 0.0001. The batch size was 16, and the epoch was 100. We used early stopping with a patience of 30 to stop training. We used random rotation and horizontal and vertical flipping for augmentation. We set the hyperparameter $\gamma$, which controls the slope of the weighted focal loss function, to 0 for 0 to 1 % and 2 for the other sections.

For proportion estimation, we used 606 core images, where the proportion interval is labeled for each core; either ‘0 to 0.01’, ‘0.01 to 0.25’, ‘0.25 to 0.5’, ‘0.5 to 0.75’, or ‘0.75 to 1.00’. We resized the images to 2048 $\times$ 2048 to input the network. Fig. 4 shows examples of core images. The images gradually turn brown along with a larger proportion of PD-L1 positive cells. However, it is necessary to observe whether the brown cells are tumors and whether the membranes are dyed. This means that it is not so easy for color-based methods. We performed 4-fold cross-validation and evaluated the average performance metrics.

Table 1 shows the mean of precision, recall, and f1 score for each method. Our method achieved the best performance compared to the other methods. Positive and negative tumor cells have various appearances depending on staining properties and patient variation. Det did not work well for three-class classifications as the positive and negative tumor cells have similar shapes, as shown in Fig. 5. To achieve accurate detection, a large amount of training data is required. Comparing Prop w/o mask and Prop, Prop is better than Prop w/o mask. Since both methods use the same loss function, it shows that the tumor mask contributes to improving performance. The accuracy of ‘0 to 1 %’ in FocalProp is much worse because this loss teats all interval sections the same even though the interval length is different, as discussed in Section 2. The proposed method outperformed other mask-based methods because weighted focal loss can handle imbalances (interval and data) while maintaining discriminative ability.

Fig. 5 shows examples of estimated intermediate positive and negative feature maps, which can be used for the interpretability of the network; pathologists can understand how the AI classifies the cells to estimate the proportion. The mask (the 3rd column) shows the cell detection results of tumor cells (including both positive and negative). The feature maps show the estimation results of positive (red) or negative (blue) classification, in which the feature map has a low resolution than the original image. A masked feature map is generated by combining the mask and feature map. Regression CAM shows the activation map from the regression network, where red indicates the pixels contributing to the network output. The first and second rows are examples of successfully estimated cases, which pathologists confirmed. In these cases, both of them are stained as brown. However, the class of cells is different; positives in the 1st column and negatives in the 2nd one, which is defined by the staining pattern even though they are brown. Our method successfully classified such difficult cases. The third row is an example of a miss-classified case, which is a difficult case. Actually, all of them are positive because their membrane is slightly stained by light brown. However, our method miss-classified them as negative. This is a difficult case, even for medical doctors. The CAM is meaningless in this task because all tumor regions are used to calculate the proportions.

We propose a proportion estimation method that can estimate the partial proportion (about only tumor cells) by using a tumor mask and address the imbalanced (interval and data) issues by our weighted focal proportion loss. We first detect tumor cells and generate a tumor mask. Then, we estimate the PD-L1 tumor proportion among tumor cells. By applying the mask, we could represent intermediate output for PD-L1 positive and negative, in which the visualization is useful for pathologists in clinical. In the experiments, our method outperforms other comparisons and achieves state-of-the-art performance.