One-Vote Veto: Semi-Supervised Learning for Low-Shot Glaucoma Diagnosis

As illustrated in Fig. 1, conventional semi-supervised learning methods initialize a network by pre-training it with a small number of fundus images for glaucoma diagnosis. However, we observed that such approaches are highly sensitive to noise. As a result, we design a novel MTSN specifically for our semi-supervised learning approach, which requires not only predicting the label of a given fundus image but also determining the similarity between a pair of given fundus images to generate pseudo labels through a voting process.

Conventional Siamese networks have become a common choice for metric learning and few/low-shot image recognition tasks [53]. These networks comprise two identical sub-networks, as depicted in Fig. 2. Each pair of fundus images $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ are separately fed into these sub-networks, producing two 1D embeddings (features) $\mathbf{h}_{i}$ and $\mathbf{h}_{j}$, respectively. Another 1D embedding $\mathbf{h}_{i,j}$ is generated by $\Phi(\cdot)$. $\mathbf{h}_{i,j}$ is then passed through a fully connected (FC) layer to produce a scalar $q(\mathbf{x}_{i},\mathbf{x}_{j})\in[0,1]$ indicating the similarity between $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$. If $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ are dissimilar, $q(\mathbf{x}_{i},\mathbf{x}_{j})$ approaches 1, and vice versa. The ground-truth labels of $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ are represented by $y_{i}\in\{0,1\}$ and $y_{j}\in\{0,1\}$, respectively, where 0 denotes a healthy image, and 1 denotes a GON image.

However, a conventional Siamese network can only determine whether $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ belong to the same category, rather than predicting their independent categories. A straightforward solution is to connect $\mathbf{h}_{i}$ and $\mathbf{h}_{j}$ to separate FC layers, producing two scalars $p(\mathbf{x}_{i})$ and $p(\mathbf{x}_{j})$ indicating the probabilities that $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ are GON images, respectively. Refer to Fig. 2 and note that the two FC layers connected to $\mathbf{h}_{i}$ and $\mathbf{h}_{j}$ use the same weights. In this article, we refer to the network architecture in Fig. 2 as an MTSN, which can simultaneously measure the similarity of a given pair of fundus images and classify them as either healthy or GON. These two tasks are dependent yet not directly deducible from one another. A well-trained glaucoma diagnosis network can be employed to compare differences between given pairs of fundus images, but a well-trained fundus image similarity measurement network cannot directly output the category of a given fundus image.

In addition, the visual features learned from the primary and secondary tasks are distinct from one another. For the primary task, the network learns the visual features to classify same-class and different-class fundus image pairs. On the other hand, for the secondary task, the network learns the visual features to classify GON and healthy fundus images. Although this network architecture may not provide complementary information, it effectively performs a type of “data augmentation” by producing $C(N,2)$ pairs of fundus images for training, rather than using $N$ independent fundus images. The visual features learned from these two tasks prove to be more informative for glaucoma diagnosis when the training set is small. Furthermore, multi-task learning is effective because requiring an algorithm to perform well on a related task induces regularization, which can be superior to uniform complexity penalization for preventing over-fitting. This idea has been explored in many Siamese neural network works, such as [54, 55, 56].

In this article, we use $n_{0}$ and $n_{1}$ to respectively denote the numbers of healthy and GON fundus images used to train the MTSN, with $n=n_{0}+n_{1}$. $n_{0}$ is usually much greater than $n_{1}$, because there are fewer patients with glaucomatous disease than healthy patients, resulting in a severely imbalanced dataset. Therefore, the MTSN is trained by minimizing a CWCE loss as follows:

where

The hyper-parameter $\lambda$ balances the primary task loss $\mathcal{L}_{\text{sim}}$ and the secondary task loss $\mathcal{L}_{\text{cla}}$. The choice of $\lambda$ and $\Phi(\cdot)$ is discussed in Sect. 4.2. The motivations for using such a CWCE loss function instead of the commonly used triplet loss [57] or contrastive loss [58] to train the MTSN are:

1. 1.

   Most datasets for rare disease diagnosis are imbalanced. As detailed in Sect. 4.1, the OHTS training set is severely imbalanced, with 50,208 healthy images and only 2,416 GON images for supervised learning, and 995 healthy images and 152 GON images for low-shot learning. Learning from such an imbalanced dataset without weights on different classes can result in many incorrect predictions, with most GON images likely to be predicted as healthy images. To address this issue, a higher weight should be used for the minority class to prevent the CNN from predicting all fundus images as the majority class.

2. 2.

   In multi-task learning, weighing different types of losses, such as regression and classification, is typically challenging [59]. Assigning an incorrect weight may cause one task to perform poorly, even when other tasks converge to satisfactory results. Therefore, formulating $\mathcal{L}_{\text{sim}}$ as a weighted cross-entropy loss function is a simple but effective solution. However, due to the dataset imbalance problem, the cross-entropy losses have to be weighted.

3. 3.

   As shown in Fig. 3, OVV self-training requires both labels and probabilities (of being GON images), predicted by a pre-trained model, to produce pseudo labels for unlabeled data. Such network architecture and training loss can efficiently and effectively provide both “self-predicted” and “contrastively-predicted” labels and probabilities, as described in Sect. 3.2.

It should be noted here that using the fundus images from the same patient as an image pair for MTSN training is not necessary.

As discussed in Sect. 1, self-training aims to improve the performance of a pre-trained model by incorporating reliable predictions of the unlabeled data to obtain useful additional information that can be used for model fine-tuning. A feasible strategy to determine such reliable predictions is, therefore, key to the success of self-training [60].

In conventional semi-supervised learning algorithms, a pre-trained image classification model (obtained through supervised learning) can be fine-tuned by assessing the reliability of unlabeled images. To determine the reliability of an unlabeled image, its probability distribution for the most likely class is compared to a pre-determined threshold. If the probability surpasses this threshold, the prediction is considered a pseudo label. Subsequently, the image and its corresponding pseudo label are utilized to fine-tune the pre-trained model.

However, relying solely on probability distributions to generate pseudo labels is often insufficient [50]. Drawing inspiration from LwEM [61], we introduce One-Vote Veto self-training in this paper, as illustrated in Fig. 3¹¹1The superscripts $r$ and $t$ denote “reference” and “target”, respectively.. Similar to LwEM [61], we use a collection of $m$ reference (labeled) fundus images $\{\mathbf{x}^{r}_{1},\dots,\mathbf{x}^{r}_{m}\}\in\mathscr{X}^{r}$ to provide “contrastive predictions” to the target (unlabeled) fundus images $\{\mathbf{x}^{t}_{1},\dots,\mathbf{x}^{t}_{m}\}\in\mathscr{X}^{t}$. The contrastive predictions subsequently vote to veto the unreliable “self-predictions” $\{\tilde{y}^{t}_{1},\dots,\tilde{y}^{t}_{m}\}$ produced by the MTSN. Our OVV self-training is detailed in Algorithm 1, where the target model updates its parameters during self-training but the reference model does not.

When fine-tuning an MTSN pre-trained through low-shot learning, each mini-batch contains a discrete set of $m$ reference fundus images $\{\mathbf{x}^{r}_{1},\dots,\mathbf{x}^{r}_{m}\}\in\mathscr{X}^{r}$, their ground-truth labels $\{{y}^{r}_{1},\dots,{y}^{r}_{m}\}\in\mathscr{Y}^{r}$, and an equal number of $m$ target fundus images $\{\mathbf{x}^{t}_{1},\dots,\mathbf{x}^{t}_{m}\}\in\mathscr{X}^{t}$ without labels. $\mathbf{h}^{r}_{k}$ and $\mathbf{h}^{t}_{k}$ represent the 1D embeddings learned from $\mathbf{x}^{r}_{k}$ and $\mathbf{x}^{t}_{k}$ ($k\in[1,m]\cap\mathbb{Z}$), respectively. Given a pair of reference and target fundus images, $\mathbf{x}^{r}_{l}$ and $\mathbf{x}^{t}_{k}$, the pre-trained MTSN can “self-predict”:

• •

  the scalars ${p}(\mathbf{x}^{r}_{l})$ and ${p}(\mathbf{x}^{t}_{k})$ which indicate the probabilities that $\mathbf{x}^{r}_{l}$ and $\mathbf{x}^{t}_{k}$ are GON images, respectively;

• •

  their labels $\tilde{y}^{r}_{l}=\delta({{p}(\mathbf{x}^{r}_{l})})$ and $\tilde{y}^{t}_{k}=\delta({{p}(\mathbf{x}^{t}_{k})})$ using its fundus image classification functionality ($\delta(p)=1$ when $p>0.5$, and $\delta(p)=0$ otherwise).

${p}(\mathbf{x}^{r}_{l})$ is then used to determine whether the reference fundus image $\mathbf{x}^{r}_{l}$ is qualified to veto unreliable predictions. If $|{p}(\mathbf{x}^{r}_{l})-{y}^{r}_{l}|>\kappa_{2}$, its vote will be omitted, where $\kappa_{2}$ is a threshold used to select qualified reference fundus images (step 6 in Algorithm 1). In the meantime, the pre-trained MTSN can also “contrastively-predict” the scalar

indicating the GON probability as well as the label

of $\mathbf{x}^{t}_{k}$ from $\mathbf{x}^{r}_{l}$ using its input similarity measurement functionality²²2In rare cases, $\tilde{y}^{r\rightarrow t}_{l,k}(\mathbf{x}^{r}_{l},\mathbf{x}^{t}_{k})$ might not be equivalent to $\delta({p}_{l,k}^{r\rightarrow t}(\mathbf{x}^{r}_{l},\mathbf{x}^{t}_{k}))$.. To determine the reliability of $\tilde{y}^{t}_{k}$ and whether it can be used as the pseudo label of $\mathbf{x}^{t}_{k}$, all the reference fundus images $\{\mathbf{x}^{r}_{1},\dots,\mathbf{x}^{r}_{m}\}\in\mathscr{X}^{r}$ in the mini-batch are used to provide additional judgements. Each pair of contrastively-predicted scalar (indicating GON probability) and label form a vote $({p}_{l,k}^{r\rightarrow t}(\mathbf{x}^{r}_{l},\mathbf{x}^{t}_{k}),\tilde{y}^{r\rightarrow t}_{l,k}(\mathbf{x}^{r}_{l},\mathbf{x}^{t}_{k}))$. With all votes collected from the qualified reference fundus images, the OVV self-training algorithm determines whether $\tilde{y}^{t}_{k}$ should be used as the pseudo label for $\mathbf{x}^{t}_{k}$ based on the following criteria (step 9 in Algorithm 1):

• •

  Identical to the process of determining qualified reference fundus images, if any ${p}_{l,k}^{r\rightarrow t}(\mathbf{x}^{r}_{l},\mathbf{x}^{t}_{k})$ ($l\in[1,m]\cap\mathbb{Z}$) or ${p}(\mathbf{x}_{k}^{t})$ is not close to either 0 (healthy) or 1 (GON), as evaluated by the threshold $\kappa_{2}$, $\tilde{y}^{t}_{k}$ will not be assigned to $\mathbf{x}^{t}_{k}$.

• •

  If a minority of more than $\kappa_{1}$ qualified reference fundus images disagree with the majority of the qualified reference fundus images, $\tilde{y}^{t}_{k}$ will not be assigned to $\mathbf{x}^{t}_{k}$.

As discussed in Sect. 4, $\kappa_{1}=0$ (all qualified reference images vote for the same category) achieves the best overall performance. Therefore, the aforementioned strategy is named “One-Vote Veto” in this paper. Since each target fundus image is required to be compared with all the reference fundus images in the same mini-batch, the proposed self-training strategy has a computational complexity of $\mathscr{O}(n^{2})$, which is relatively memory-consuming. The reliable target fundus images and their pseudo labels are then included into the low-shot training data to fine-tune the pre-trained MTSN with supervised learning by minimizing a CWCE loss. The OVV self-training performance with respect to different $\kappa_{1}$, $\kappa_{2}$, and $m$ values is discussed in Sect. 4.

The datasets utilized in our experiments were collected by various clinicians in different countries using distinct fundus cameras. The ACRIMA [22] and LAG [9] datasets are publicly available, while the OHTS [11, 12] and DIGS/ADAGES [23] datasets are available upon request after appropriate data use agreements are initiated. Their details are as follows:

• •

  The OHTS [11, 12] is the only multi-center longitudinal study that has precise information on the dates/timing of the development of glaucoma (the enrolled subjects did not have glaucoma at study entry) using standardized assessment criteria by an independent Optic Disc Reading Center and confirmed by three glaucoma specialist endpoint committee members. In our experiments, a square region centered on the optic nerve head was first extracted from each raw fundus image using a well-trained DeepLabv3+ [63] model. A small part of the raw data are stereoscopic fundus images, each of which was split to produce two individual fundus images. Through this image pre-processing approach, a total of 74,678 fundus images were obtained. Moreover, ENPOAGDISC (endpoint committee attributable to primary open angle glaucoma based on optic disc changes from photographs) [3] labels are used as the classification ground truth. The fundus images are divided into a training set (50,208 healthy images and 2,416 GON images), a validation set (7,188 healthy images and 426 GON images), and a test set (13,780 healthy images and 660 GON images) by participant. Splitting by participant (instead of by image) ensured that the validation and test sets did not contain images from any eyes or individuals used to train the model. More details on dataset preparation and baseline supervised learning experiments are provided in our recent publications [3, 10]. Additionally, we select one image (from only one eye) from each patient in the training set to create the low-shot training set (995 healthy images and 152 GON images). If both eyes of a patient do not convert to glaucoma in the study, the first captured fundus photograph is selected. If either eye of a patient converts to glaucoma in the study, the first glaucoma fundus photograph is selected.

• •

  The ACRIMA [22] dataset consists of 309 healthy images and 396 GON images. It was collected as part of an initiative by the government of Spain. Classification was based on the review by a single experienced glaucoma expert. Images were excluded if they did not provide a clear view of the optic nerve head region [43].

• •

  The LAG [9] dataset contains 3,143 healthy images and 1,711 GON images³³3The number of fundus images being published is fewer than what was reported in publication [9]. , collected by Beijing Tongren Hospital. Similar to the OHTS dataset, we also use the well-trained DeepLabv3+ [63] model to extract a square region centered on the optic nerve head from each fundus image.

• •

  The DIGS and ADAGES [23] are longitudinal studies designed to detect and monitor glaucoma based on optical imaging and visual function testing that, when combined, have generated tens of thousands of test results from over 4,000 healthy, glaucoma suspect, or glaucoma eyes. In our experiments, we utilize the DIGS/ADAGES test set (5,184 healthy images and 4,289 GON images) to evaluate the generalizability of our proposed methods.

Visualizations of the four test sets using t-SNE [62] are provided in Fig. 4. Since healthy and GON images are distributed similarly between the OHTS and LAG datasets, we expect models to perform similarly on these datasets. Dissimilar distributions in the ACRIMA and DIGS/ADAGES datasets led us to believe the performance of models on these datasets would be somewhat worse. Using these four datasets, we conduct three experiments:

1. 1.

   Supervised learning experiment: We employ transfer learning [64] to train ResNet-50 [38], MobileNet-v2 [46], DenseNet [65], and EfficientNet [66] (pre-trained on the ImageNet database [37]), on the entire OHTS training set (including $\sim$53K fundus images). The best-performing models are selected using the OHTS validation set. Their performance is subsequently evaluated on the OHTS test set, the ACRIMA dataset, the LAG dataset, and the DIGS/ADAGES test set.

2. 2.

   Low-shot learning experiment: The four pre-trained models mentioned above are used as the MTSN backbones and trained on the OHTS low-shot training set (containing 1,147 images) to validate the effectiveness of our proposed low-shot glaucoma diagnosis algorithm. The validation and testing procedures are identical to those in the supervised learning experiment.

3. 3.

   Semi-supervised learning experiment: The MTSNs trained on the low-shot training set are fine-tuned on the entire OHTS training set without using additional ground-truth labels. The fine-tuned MTSNs are referred to as MTSN+OVV. The validation and testing procedures are identical to those in the supervised learning experiment.

The fundus images are resized to $224\times 224$ pixels. The initial learning rate is set to 0.001, which decays gradually after the 100th epoch. Due to the dataset imbalance problem, F1-score is utilized to select the best-performing models during the validation stage. Moreover, we adopt an early stopping mechanism during the validation stage to reduce over-fitting (the training will be terminated if the achieved F1-score has not increased for 10 epochs). We use three metrics: (1) accuracy, (2) F1-score, and (3) AUROC to quantify the performances of the trained models. While accuracy is generally reported in image classification papers, F1-score and AUROC are more comprehensive and reasonable evaluation metrics when the dataset is severely imbalanced.

We set $\lambda$ in (1) to 0.1, 0.2, 0.3, 0.4, and 0.5, respectively, and compare the MTSN performance when $\Phi$ computes the element-wise absolute difference (EWAD) and element-wise squared difference (EWSD), respectively. The comparisons in terms of F1-score and AUROC on the OHTS test set are provided in Fig. 5. It can be seen that the MTSN achieves the best overall performance when $\lambda=0.3$. This is reasonable, as a higher $\lambda$ weighs more on the image classification task, easily resulting in over-fitting. Additionally, the MTSN in which $\Phi(\cdot)$ computes the EWSD between $\mathbf{h}_{i}$ and $\mathbf{h}_{j}$ performs better when using ResNet-50 as the backbone CNN but slightly worse when using MobileNet-v2 as the backbone CNN. Therefore, we further evaluate their generalizability on three additional test sets, as shown in Table 1. When $\Phi(\cdot)$ computes the EWAD, the MTSN generally performs better or very similarly on the additional test sets, especially when testing the MTSN assembled with ResNet-50 on the ACRIMA dataset. EWAD is, therefore, used in the following experiments. Furthermore, Table 1 provides the results of a baseline supervised learning experiment conducted on the low-shot training set. The results suggest that low-shot learning performs much better than supervised learning when the training size is small.

Furthermore, we discuss the selection of the thresholds $\kappa_{1}$ and $\kappa_{2}$ (used to select reliable “self-predictions” and “contrastive predictions” in our OVV self-training) as well as the impact of different mini-batch sizes $2m$ on OVV self-training (each mini-batch contains $m$ pairs of reference and target fundus images). Table 2 shows the MTSN performances with respect to different $\kappa_{1}$, $\kappa_{2}$, and $m$. When evaluated on the OHTS test set, it can be seen that accuracy and F1-score increase slightly, but AUROC almost remains the same, with the decrease of $m$. Moreover, with the increase of $\kappa_{1}$ and $\kappa_{2}$, the standard to determine reliable predictions becomes lower, making the semi-supervised learning performance degrade. Based on this experiment, we believe OVV self-training benefits from smaller $\kappa_{1}$ and $\kappa_{2}$.

Additionally, MTSNs, trained under different $m$, are evaluated on the three additional test sets, as shown in Table 2. It can be seen that the network trained with a larger $m$ typically shows better results. When $m$ decreases, the generalizability of MTSN degrades dramatically, especially for F1-score (decreases by around 9-19%). Therefore, increasing the mini-batch size can improve the MTSN generalizability, as more reference fundus images are used to provide contrastive predictions for the target fundus images, which can veto more unreliable predictions on the unlabeled data. Hence, we increase $m$ to 30 to further improve OVV self-training when comparing it with other published SoTA algorithms, as shown in Sect. 4.4. Since our threshold selection experiments cover a very limited number of discrete sets of $\kappa_{1}$, $\kappa_{2}$, and $m$, we believe better performance can be achieved when more values are tested.

Comparisons of supervised learning, low-shot learning, and semi-supervised learning (w.r.t. four backbone CNNs: ResNet-50, MobileNet-v2, DenseNet, and EfficientNet) for glaucoma diagnosis are provided in Table 3. First, these results suggest that the MTSNs fine-tuned with OVV self-training that requires a small number of labeled fundus images perform similarly (AUROC 95% CI overlaps considerably) and, in some cases, significantly better (AUROC 95% CI does not overlap) than the backbone CNNs trained with a large number of labeled fundus images (50 times larger) under full supervision.

Specifically, when using ResNet-50, MobileNet-v2, or DenseNet as the backbone CNN, semi-supervised learning performs similarly to supervised learning on the OHTS and DIGS/ADAGES test sets, and in most cases, significantly better than supervised learning on the ACRIMA and LAG datasets. Although EfficientNet trained through supervised learning performs unsatisfactorily on all four test sets, it shows considerable compatibility with MTSN in the low-shot and semi-supervised learning experiments. Second as expected, the AUROC scores achieved by low-shot learning are in most, but not all, cases slightly lower than those achieved by the backbone CNNs, when evaluated on the OHTS test set. However, low-shot learning shows better generalizability than supervised learning on the ACRIMA and LAG datasets. Moreover, since low-shot learning uses only a small amount of training data, training an MTSN is much faster than supervised learning. As MTSNs assembled with ResNet-50 and MobileNet-v2 typically demonstrate better performances than the ones assembled with DenseNet and EfficientNet, we only use the former two CNNs for the following experiments.

We also employ Grad-CAM++ [67] to explain the models’ decision-making, as shown in Fig. 6. These results suggest that the optic nerve head areas impact model decisions most. The neuroretinal rim areas are identified as most important, and the periphery contributed comparatively little to model decisions for both healthy and GON eyes [42].

We also carry out a series of experiments with respect to different percentages of training data, as shown in Table 4(b), to further validate the effectiveness of our proposed low-shot and semi-supervised learning algorithms. The backbone CNNs trained via supervised learning on the small subsets generally perform worse than the MTSNs trained via low-shot and semi-supervised learning. With the labeled training data increase, the models’ performance gets saturated. When using less labeled training data ($0.5\%$ and $1.0\%$), the MTSN performance degrades. However, its performance can still be greatly improved with OVV self-training (accuracy, F1-score, and AUROC can be improved by up to 6%, 11%, and 0.08, respectively). In addition, when using over $10\%$ of the entire training data, the MTSN performance saturates, and the OVV self-training can bring very limited improvements on MTSNs.

Table 5 provides comprehensive comparisons with nine SoTA glaucoma diagnosis algorithms⁴⁴4Our recent work [3] provides the baseline supervised learning results.. The results suggest that (a) for low-shot learning, the MTSNs trained by minimizing our proposed CWCE loss perform significantly better than the SoTA low-shot glaucoma diagnosis approach [19] on all four datasets (accuracy, F1-score, and AUROC are up to 5%, 15%, and 0.08 higher, respectively), and (b) for semi-supervised learning, the MTSNs fine-tuned with OVV self-training also achieve the superior performances over another two SoTA semi-supervised glaucoma diagnosis approaches [20, 21] (accuracy, F1-score, and AUROC are up to 6%, 11%, and 0.07 higher, respectively). Compared with the SoTA supervised approaches, the fine-tuned MTSNs demonstrate similar performance on the OHTS test set and better generalizability on three additional test sets. Therefore, we believe that MTSN with our proposed OVV self-training is an effective technique for semi-supervised glaucoma diagnosis.

Table 6 provides a comprehensive comparison of our proposed OVV self-training approach with four SoTA general-purpose semi-supervised learning methods: FreeMatch [24], SoftMatch [25], FixMatch [26], and FlexMatch [27], which all employ vision Transformer [28] as their backbone network. Our results demonstrate that the proposed OVV self-training approach outperforms these methods in terms of F1-score and AUROC on the OHTS dataset. Specifically, we observe improvements in F1-score ranging from approximately 11% to 13%, and improvements in AUROC ranging from around 9% to 20%. Furthermore, our method demonstrates better generalizability in terms of AUROC across three additional fundus image test sets. Although their results are inferior to ours, particularly in terms of AUROC, it may be unjust to compare them as they were not specifically designed for the diagnosis of glaucoma or other diseases.

We conduct two additional few-shot multi-class lung disease diagnosis experiments: (a) chest X-ray image classification for COVID-19 and viral pneumonia detection [29, 30], and (b) lung histopathological image classification for lung cancer diagnosis [31], to validate the effectiveness of our proposed MTSN and CWCE loss. The first experiment has three classes of images: (1) healthy, (2) viral pneumonia, and (3) COVID-19 (an example of each class is shown in Fig. 7), while the second experiment also has three classes of images: (1) benign tissue, (2) adenocarcinoma, and (3) squamous cell carcinoma (an example of each class is shown in Fig. 7). In these two experiments, we only select a few images from each class for MTSN training. The numbers of images used for training, validation, and testing are given in Table 7, where it can be observed that the training set is much smaller than the validation and test sets. Since (1) can only be used for binary image classification problems, we extend it here to tackle multi-class image classification problems.

Let us consider the chest X-ray image classification task as an example. Each image $\mathbf{x}$ is assigned a pair of two labels $(r,s)$ with the following values:

• •

  $r=1$ and $s=0$ when $\mathbf{x}$ is a healthy image;

• •

  $r=2$ and $s=1$ when $\mathbf{x}$ is a viral pneumonia image;

• •

  $r=3$ and $s=3$ when $\mathbf{x}$ is a COVID-19 image.

The numbers of healthy images (class 1), viral pneumonia images (class 2), and COVID-19 images (class 3) are denoted as $n_{1}$, $n_{2}$, and $n_{3}$, respectively. The total number of images is $N=n_{1}+n_{2}+n_{3}$. The weight $\omega{{}_{\text{cla},e}}$ used in the image classification loss $\mathcal{L}{{}_{\text{cla}}}$ w.r.t. class $e$ is given by:

$\sum_{e=1}^{3}\omega{{}_{\text{cla},e}}=1$. Therefore, $\mathcal{L}{{}_{\text{cla}}}$ can be written as follows:

where $p_{e}(\mathbf{x})\in[0,1]$ indicates the probability that $\mathbf{x}$ belongs to class $e$. $\sum_{e=1}^{3}p_{e}(\mathbf{x})=1$. $k_{e,i}=1$ when $e=r_{i}$, and $k_{e,i}=0$ otherwise. Given a pair of images $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ with ground-truth labels $(r_{i},s_{i})$ and $(r_{j},s_{j})$, respectively, there are four cases:

• •

  case 1: $|s_{i}-s_{j}|=1$ (healthy v.s. viral pneumonia);

• •

  case 2: $|s_{i}-s_{j}|=2$ (viral pneumonia v.s. COVID-19);

• •

  case 3: $|s_{i}-s_{j}|=3$ (COVID-19 v.s. healthy);

• •

  case 4: $|s_{i}-s_{j}|=0$ (two images are of the same class).

The weight $\omega{{}_{\text{sim},c}}$ used in the image similarity comparison loss $\mathcal{L}{{}_{\text{sim}}}$ with respect to case $c$ is given by:

$\sum_{c=1}^{4}\omega{{}_{\text{sim},c}}=1$. Therefore, $\mathcal{L}{{}_{\text{sim}}}$ can be written as follows:

where $q_{c}(\mathbf{x}_{i},\mathbf{x}_{j})\in[0,1]$ indicates the similarity between $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ under case $c$. $\sum_{c=1}^{4}q_{c}(\mathbf{x}_{i},\mathbf{x}_{j})=1$. $h_{c,(i,j)}=1$ when $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ belong to case $c$, and $h_{c,(i,j)}=0$ otherwise. The hyper-parameter $\lambda$ is empirically set to $0.3$.

The experimental results of these two multi-task biomedical image classification tasks are presented in Fig. 8 with two confusion matrices. These results demonstrate that our proposed MTSN can be effectively trained with very few images to solve multi-class biomedical image classification problems. Specifically, the achieved accuracy values for chest X-ray image classification ($\sim$25-shot learning) and lung histopathological image classification (10-shot learning) are 93% and 90%, respectively. The chest X-ray image classification result compares favorably with the accuracy range of 82%-93% achieved by supervised methods (2,520 images for training, 840 images for validation, and 840 images for testing) using all available training data [68]. Although the accuracy achieved by MTSN for lung histopathological image classification is lower than the accuracy of over 97% reported in [69] by supervised approaches using the full training set (8,250 images for training, $\sim$3,000 images for validation, and 3,744 images for testing), we believe that our proposed low-shot learning method can achieve comparable results when a small amount of additional images are incorporated for MTSN training.