Hierarchical Knowledge Guided Learning for Real-world Retinal Diseases Recognition

The computer-assisted diagnosis (CAD) methods for retinal diseases such as DR, Glaucoma and age-related macular degeneration (AMD) have been long recognized. Gulshan et al. [2] and Gargeya et al. [7] proposed to use a CNN for binary classification of with/without DR. Zhang et al. [20] presented an ensemble strategy to perform two-class and four-class classifications. Some methods [21, 22, 23] proposed to combine the segmentation results and grading results to provide the interpretability with the internal correlation between DR levels and lesions. Besides the OC/OD area-based method [6] mentioned above, there are some previous works that generate evidence maps for glaucoma diagnosis. Zhao et al. [24] proposed a two-stage cascaded approach that obtains unsupervised feature representation of fundus image with a CNN and cup-disc-ratio value regression by random forest regressor. The automated diagnosis of AMD has also been studied [25, 26]. Ju et al. [27] leveraged the common features between DR and AMD, then improve both accuracy under a knowledge distillation and multi-task manner.

Although some specific retinal diseases are fully studied, the condition of the fundus is complex. Most of the existing methods do not promise the robustness and ability to diagnose other diseases, especially rare ones. Kaggle EyePACS dataset [28] consists of 35,126 training images graded into five DR stages. However, it is found that there are at least more than 30 kinds of retinal diseases that are mislabeled by the original annotators [10]. Ignoring those out-of-distribution categories leads to a huge risk. Wang et al. [13] first used a multi-task framework to detect 36 kinds of retinal diseases, respectively. However, it requires extra annotations for the location of the optic disc and macula area. Quellec et al. [14] also took the features among diseases into consideration and leveraged the few-shot learning technique to perform rare pathologies recognition. The existing methods are inspiring but show some limitations: those methods consider the prior knowledge of retinal diseases, such as the common features (lesions) and locations shared by some diseases, but still lack direct insights for leveraging some prior knowledge or cognitive laws.

We group existing methods for long-tailed learning into three main categories based on their main technical contributions, i.e., re-sampling, re-weighting, and feature sharing.

Re-smapling methods try to balance the distribution by over-sampling the minority-class samples or under-sampling the majority-class samples. Kang et al. [19] proposed a two-stage training strategy to train the feature extractor and classifier from a uniform and re-sampling distribution, respectively. Zhou et al. [18] proposed to design a two-branch network to achieve one-stage training with the same idea. Wang et al. [29] presented a curriculum learning-based sampling scheduler. Most of these works focus on single-label datasets. For multi-label datasets, over-sampling of minority class will sometimes sample the majority class at the same time due to the label co-occurrence and therefore leads to a new imbalanced condition. To handle this challenge, Wu et al. [16] extended the re-sampling strategy into a multi-label scenario and introduced a regularization term to overcome the over-suppression from negative samples.

Re-weighting methods aim to design robust loss functions for learning from imbalanced data. Focal loss [30] computes weight for each training sample and achieves Hard Samples Mining according to the prediction probabilities. The increase in the sample brings about a diminishing return on performance for those majority classes. Cui et al. [31] proposed to assign a new variable to increase the benefit from those effective samples. Cao et al. [32] proposed an effective training strategy, which allows the model to learn an initial representation while avoiding some of the complications associated with re-weighting or re-sampling.

Feature Sharing methods aim to learn the general knowledge from the majority classes and then transfer it to the minority classes. Liu et al. [33] proposed OLTR, which learns a set of dynamic meta embedding to transfer the visual information knowledge of the head to the tail category. Also, they designed a memory set that allows tail categories to utilize relevant head information by calculating the similarity. Xiang et al. [34] found that learning from a less-imbalanced subset suffers from little performance loss and proposed to train multiple shot-based teacher models, then guide the training of a unified student model. Li et al. [35] proposed balanced group softmax (BAGS) to improve long-tailed object detection by grouping the categories with similar numbers of training instances. BAGS implicitly modulates the head and tail classes are sufficiently trained, without requiring any extra sampling for the tail classes.

To address the challenge of long-tailed retinal diseases recognition, we evaluate the proposed methods on two in-house and two public datasets with different scales, including the number of samples, the number of classes, the imbalance ratio, and the degree of label co-occurrence, e.g., label cardinality.

Here, we give the quantitative metrics for the main characteristics of long-tailed retinal disease recognition. The first is how imbalanced the original distribution is. Formally, for the original distribution $S=\{(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{n},y_{n})\}$ where each $x$ indicates a feature vector and each $y$ is an associated one-hot label. The indexes of k kinds of categories are sorted from most to least with $N_{c_{1}}>N_{c_{2}}>...>N_{c_{k}}$ where $N_{c_{*}}$ denotes the number of instances of the $*_{th}$ category. Thus, the imbalance ratio can be simply described as $\rho=\frac{N_{c_{1}}}{N_{c_{k}}}$. Moreover, as we claimed that there is label co-occurrence among the original long-tailed distribution, and most re-sampling strategies will lead to a new inner-class imbalance [16] (we have a detailed analysis of this in Sec. 6). Here, we introduce several useful indicators for measuring the label-occurrence [36, 37]. The first is label cardinality which indicates the average number of labels per example: $L_{Card}(S)=\frac{1}{n}\Sigma^{n}_{i=1}|y_{i}|$. The characteristics of the four datasets are shown in Table I.

Our collected private datasets were acquired from private hospitals over the time span of 10 years. Some commonly used datasets are also included, such as ODIR and RFMiD. Each image was labeled or relabeled by 3 - 8 senior ophthalmologists. A sample is retained only if more than half of the ophthalmologists are in agreement with the disease label, or the sample will be assigned to re-labeled processing with the discussion of all ophthalmologists. In this study, more than one million samples are selected to form the two datasets, Retina-100K and Retina-1M, which consist of more than 50 kinds of retinal diseases. The Retina-100 K dataset is randomly sampled from the Retina-1M dataset, some rare diseases with no more than 10 instances are not sampled, resulting in the number of categories being only 48. It should be noted that some low-quality fundus images are potentially at risk of being misdiagnosed as cataracts. Therefore, a group of quality measurement categories is also added. In addition, we found that some non-fundus images could also be mistakenly predicted as retinal diseases during the inference stage. Similarly, we added a category for non-fundus images to reduce this risk.

We would like to emphasize that this is the first study that attempts to train the DL-based model for retinal disease recognition from a database with more than one million samples. It can be found that the Retina dataset exhibits an extreme long-tailed distribution with $\rho$ closing to 80,000. It is noticed that, with the increase in the number of samples, the samples of various categories are also enriched, and the label co-occurrence is reduced (e.g., 1.3439 for Retina-100K and 1.5046 for Retina-1M in terms of $L_{Card}$.) However, there are also some samples that exhibit a high label co-occurrence (e.g., it is counted that 5,326 samples exist more than five diseases in Retina-1M).

The sorted classes are divided into many, medium, and few groups. The cut-off points and the number of samples ratio are many / medium / few $\approx$ 100 / 50 / 10 for Retina-100K; many / medium / few $\approx$ 1000 / 200 / 10 for Retina-1M.

Like the two public datasets mentioned above, all diseases are mapped into a semantic hierarchical tree. Specifically, we regard the 50+ kinds of diseases as the base/finest category, and we divide them into several groups according to their characteristics. There are three hierarchical levels for each base category in total, as shown in Fig. 2. For instance, a sample is labeled as big drusen, and it also belongs to drusen and AMD. There are also three other base categories in the drusen subset: small drusen, medium drusen, and non-macular drusen. For a better overview of how our hierarchies are conducted and further reproducibility, we give a detailed description of the group generation for different hierarchies in our supplementary files.

Suppose the original data distribution $\mathcal{S}=\{(\mathcal{X},\mathcal{Y})\}$, where $\mathcal{X}=\{x_{1},x_{2},...,x_{N}\}$ are the N instances, and $\mathcal{Y}=\{y_{1},y_{2},...,y_{N}\}$ denotes the associated labels. Specially, for a multi-label setting, each sample $\textbf{y}_{i}=[y_{c_{1}},y_{c_{2}},...,y_{c_{k}}]$, where $k$ denotes the total number of possible categories and $\textbf{y}_{i}\in\{0,1\}^{k}$, i.e., $y_{c_{j}}$ = 1 indicates the presence of the category $c_{j}$ in image $x_{i}$ and $y_{c_{j}}$ = 0 otherwise. All indexes of category are sorted into $\{c_{1},c_{2},...,c_{k}\}$ in decreasing order with $N_{c_{1}}>>N_{c_{k}}$. Our goal is to improve the training of a deep neural network model (feature extractor $f(\cdot)$ and classifier $g(\cdot)$) from the long-tailed fundus database for the recognition of various retinal diseases.

The classes of objects in the real world naturally exhibit a hierarchical form, e.g., from domains to species in taxonomy. For instance, we show three datasets both of which define a hierarchy-tree structure for various categories. Here, we define the categories with the finest semantic meaning as child class, some child classes with common features or characteristics in semantics can be grouped into a coarse class, defined as parent class. For example, in COCO [40], the parent class Vehicle consists of several child classes such as bicycle and car. However, those classes share only a little semantic similarity. ETHEC [17] is a widely-used dataset for hierarchical classification. Although Pieris and colia belong to the family Pieridae, it is still difficult to connect them visually.

Compared with natural images, most retinal diseases can be divided into several subclasses as the lesions progress, such as diabetic retinopathy. Therefore, using the hierarchical information to train the retinal disease diagnosis model shows significant advantages. Different from previous work [12], which presented 3 kinds of 2-level relational subsets generation: shot-based, region-based and feature-based. In this study, after the discussion with more than 10 senior ophthalmologists, we refined the hierarchical mapping by taking both region information and feature information into consideration, as we presented in Sec. III-Datasets¹¹1We have included a detailed description of hierarchical mapping in our supplementary files.. In the following parts, we will give a detailed explanation of how to incorporate the defined hierarchical information into the training of deep neural networks.

Our main idea is to minimize the classification loss of all categories over different hierarchical levels. Assuming we have $M$ hierarchical levels, our target loss could be:

where BCE denotes the binary cross-entropy loss. A potential solution for hierarchy-aware constraints could be per-level classifiers [17]. The model explicitly outputs separate predictions per level for a given image using $\mathcal{M}$ classifiers. However, per-level classifiers lose connection across different hierarchical levels since the parameters of each classifier are optimized independently.

In this study, we introduce Multi-label Marginalization classifier (MLMC) as an extension of the marginalization classifier [17], which is shown in Fig. 5, to inject the hierarchical information of retinal diseases into the model training. Formally, a single classifier $g(\cdot)$ outputs a probability distribution over the final level (level 1) of the class hierarchy (finest classes). Instead of building classifiers for the remaining parent classes of $M-1$ levels, we compute the probability distribution over each one by summing the probability of their corresponding child classes.

Given the parent category ${c}_{k}$ in level $m+1$ and its corresponding $o_{m}$ child classes $\{c_{1},c_{2},...,c_{o_{m}}\}$ in level $m$, we can obtain its predicted probabilities for level $m$:

where $\sigma(\cdot)$ denotes the sigmoid activation function. Then, we calculate the loss of parent category ${c}_{k}$ using general BCE loss:

It should be noted that, for the prediction probabilities of the lowest hierarchy $l_{1}$, we directly use the logits from the outputs of fully-connected layer $g(f(\mathcal{X}))$:

Then, losses across all levels are summed for global optimization. Compared with per-level classifiers, although MLMC does not explicitly predict scores of parent classes, the models is still penalized for incorrect predictions across the $M$ levels. Also, as we mentioned in Sec. III-B1, for those child classes that could belong to more than one parent class, MLMC still has good generalization ability by simply summing the logits of the corresponding classes to different parent classes.

In this section, we introduce two commonly used sampling strategies in the long-tailed multi-class classification: instance-balanced sampling and class-balanced sampling. Then we give an experimental analysis to explain why these two sampling ways do not work well in a multi-label setting. Finally, we propose to use an instance-wise class-balanced sampling to handle this scenario.

Xiang et al. [34] and Ju et al. [12] have explored knowledge distillation in long-tailed classification. The original long-tailed distribution is divided into several subsets, which are in a relatively balanced status. Those subsets are used to train several teacher models, which are then distilled into a unified student model. However, there are some obvious shortcomings: (1) the divided subsets result in an incomplete distribution and limited features representation learned by the teacher model which may constraint the performance of the student model; (2) most performance improvements benefit from the knowledge distillation instead of learning from a relatively balanced subset; (3) some rare diseases can not directly be divided by independent feature-based or region-based rules proposed by [12].

Another observation is that, two-stage learning methods [19, 42] indicates that “training from imbalanced distribution produces a strong feature representation and the instance-based sampling produces a less-biased classifier.” To this end, we propose to enhance the two-stage learning and multiple knowledge distillation by introducing a novel hybrid multiple knowledge distillation method which can distill efficient information for long-tailed learning on both feature-level and classifier-level, i.e., logits-level [43].

Formally, we first trained two teacher models $\mathcal{F}_{t_{1}}=g_{t_{1}}(f_{t_{1}}(\cdot))$ and $\mathcal{F}_{t_{2}}=g_{t_{2}}(f_{t_{2}}(\cdot))$ using general classification loss (e.g., MLMC) and re-sampling classification loss (e.g., ICS-MLMC). Inspired by two-stage training [18, 19, 44] that model learns a good representation for feature extractor under an original distribution, we first leverage the feature distillation from $\mathcal{F}_{t_{1}}$ to assist the student model training. Given the student model $\mathcal{F}_{s}=g_{s}(f_{s}(\cdot))$, The feature-level knowledge distillation can be formulated as follows:

where the extracted feature $v=f(x)$, and $l_{sim}(\cdot)=1-cos(\cdot)$ to calculate the cosine distance (similarity) between output features from teacher and student models. Then, the standard KD is used to distill the knowledge from $F_{t_{2}}$ to the student model, which can be formulated as:

where $\mathcal{T}$ is the hyper-parameter for temperature scaling and $l_{KD}$ is the Kullback-Leibler divergence loss: $l_{KD}(p_{1}|p_{2})=p_{1}\cdot\mathrm{ln}\frac{p_{1}}{p_{2}}$. Hence, we have the total loss:

where $\alpha$ and $\beta$ are used to control the KD loss weights while $\gamma$ for prevent $L_{F-KD}$ from being close to zero. For the simplification of the hyper-parameters searches, we keep $\alpha=\beta$ here.

Hybrid multiple knowledge distillation enables the teacher model to train from the original distribution using a traditional knowledge distillation manner [43]. In this way, teacher models maintain the complete feature representations, to prevent the student model’s performance from being limited by itself performance bottlenecks.

We use ResNet-50 [45] with pre-trained weights from ImageNet as our backbone network. The input size is 512 × 512 for two in-house datasets and 224 × 224 for two public datasets, respectively. We apply Adam to optimize the model. The learning rate starts at $1\times 10^{-3}$ and decreases ten-fold when there is no drop in validation loss till $1\times 10^{-7}$ with the patience of 5 epochs. The $\alpha$ and $\beta$ are set as 0.2. The $\gamma$ is set as 10. We apply regular data-augmentation transformations during the training phase, such as random crop and flip. Following most long-tailed multi-label works [16, 44], we use the mean average precision (mAP) as the evaluation metric. All the results are calculated after 5-time running with random seeds and a batch size of 128. In all results presented, we report the 5-trial average performance and mean standard deviation. All experiments are implemented using the PyTorch platform and 8 × NVIDIA RTX 3090 GPUs.

In this section, we give a comprehensive comparison study on two in-house datasets with various baselines including some state-of-the-art works for long-tailed classification: (1) Empirical Risk Minimization (ERM); (2) vanilla Re-sampling (RS) ; (3) vanilla Re-weighting (RW); (4) OLTR [33]; (6) Focal Loss [30] ; (7) LDAM [32]; (8) CBLoss [31] (9) DBLoss [16]; (10) ASL [46].

The overall results are shown in Table III. Different baseline methods are grouped by their model designs, such as loss functions. The results are presented by groups many, medium and few to test how each method reacts to classes with different numbers of samples. In the following, we also use the head and tail classes to refer to group many and group medium. We report the average results of the three groups to view the global performance, which is denoted as “average”. To better understand the trade-off between different shot-based groups for different comparison methods, we mark the best and second performance in red and blue, respectively. From the results reported in Table III, among all baselines, ASL achieves the best results - 61.94% mAP of average results over three groups and 52.15% mAP over all classes on Retina-100K, followed by Focal Loss - 61.78% mAP. Moreover, we find that although RS achieves a good performance in the group of few, it brings a catastrophic impact on the many-shot classes (70.89% mAP $\rightarrow$ 65.72% mAP) and medium-shot classes (71.93% mAP $\rightarrow$ 67.17% mAP), resulting from the label co-occurrence in a multi-label setting. However, we fail to obtain a satisfactory performance in terms of CBLoss, which was originally designed for multi-class long-tailed classification.

For our framework, we first train two vanilla teacher models using different sampling strategies under hierarchy-aware constraints as teacher models, denoted by “baseline-original” and “baseline-ICS”, respectively. Then we distill the knowledge from two baseline teacher models into a unified student model under a hybrid distillation manner. Take Retina-100K for instance, the results of “baseline-original” show that the hierarchical pre-training can benefit the ERM baseline model (from 59.57 %mAP to 60.29 %mAP) without using any specific re-sampling strategies. Furthermore, the ICS can further improve the overall performance (from 60.29 %mAP to 61.67 %mAP) with a marginal performance loss in the many-shot group setting (from 71.18 %mAP to 70.67 %mAP). The last row demonstrates the superior performance of our proposed methods, which outperform all competitors on both two datasets Retina-100K and Retina-1M and the superiority holds for all metrics.

In this part, we aim to investigate the upper bound of the retinal diseases recognition model based on some empirical analysis when training from a long-tailed distribution.