Flexible Sampling for Long-tailed Skin Lesion Classification

Training from the semantic labels as supervision signal on long-tailed distribution can bring significant bias since majority classes can occupy the dominant portion of the feature space [12]. A straightforward solution is to leverage self-supervised learning (SSL) technique to obtain category distribution-agnostic features. Different from the supervised learning methods, SSL techniques do not require annotations but learn representations via maximizing the instance-wise discriminativeness [13, 11]. The contrastive learning loss [19] adopted in this study can be formulated as:

where $v_{i}^{+}$ is a positive sample for $x_{i}$ after the data augmentation, $v^{-}$ is a negative sample drawn from any other samples, $T$ is the temperature factor. SSL can learn stably well feature spaces which is robust to the underlying distribution of a dataset, especially in a strongly-biased long-tailed distribution [11].

To sample anchor points for training the inference model, we first comupte the mean features for each class as the prototype. Concretely, after pre-training a CNN model $M_{pre}$ with feature representation layer $f(\cdot)$ and classifier $h(\cdot)$, we can obtain features $[f(x_{1}),...,f(x_{i}),...,f(x_{n})]$ for n instances from original distribution $[x_{1},...,x_{i},...,x_{n}]$ by forward feeding into $f(\cdot)$. Thus, for sorted class index $[c_{0},...,c_{j},...,c_{k-1}]$ where $N_{c_{0}}>...>N_{c_{k-1}}$, the prototype $m_{c_{j}}$ of class $c_{j}$ can be calculated by averaging all the features from the same class:

where $N_{c{j}}$ denotes the number of instances of class $j$. Then, we find those representative samples which have the lowest distances from prototypes by calculating their Euclidean Distance:

By sorting the distances $DS_{c_{k}}=[ds_{1},...,ds_{N_{c_{k}}}$], we can select those top-$\hat{N}_{c_{k}}$ lowest-distance samples as the class anchor points to form the subset. For instance, given a long-tailed distribution following Pareto Distribution with the number of classes $k$ and imbalance ratio $r=\frac{N_{0}}{N_{k-1}}$, the number of samples for class $c\in[0,k)$ can be calculated as $N_{c_{j}}={(r^{-(k-1)})}^{c_{j}}*N_{0}$. Then, we use a scaling hyper-parameter $s\in(0,1)$ as the imbalance ratio of anchor points $\hat{r}=r\cdot s$ and we can sample a less-imbalanced subset with $\hat{N}_{c_{j}}={(\hat{r}^{-(k-1)})}^{c_{j}}*N_{0}$.

Using the prototype for sampling anchor points show the following two advantages: 1) training the inference model with representative anchor points provides a better initialization compared to that of using random selected samples; 2) a re-sampled subset shows a more relatively balanced distribution, which helps reducing confirmation bias for predicting uncertainty on those unsampled instances.

We evaluate the competitive baselines and our methods on two dermatology datasets from ISIC [9] with different number of classes. ISIC-2019-LT is a long-tailed version constructed from ISIC-2019 Challenge¹¹1https://challenge2019.isic-archive.com/, which aims to classify 8 kinds of diagnostic categories. We follow [18] and sample a subset from a Pareto distribution. With $k$ classes and imbalance ratio $r=\frac{N_{0}}{N_{k-1}}$, the number of samples for class $c\in[0,k)$ can be calculated as $N_{c}={(r^{-(k-1)})}^{c}*N_{0}$. For this setting, we set $r=\{100,200,500\}$. We select 50 and 100 images from the remained samples as validation set and test set. The second dataset is augmented from ISIC-2019 to increase the length of tail with extra 6 classes (14 in total) added from the ISIC Archive²²2https://www.isic-archive.com/ and named as ISIC-Archive-LT³³3Please see our supplementary document for more details.. This dataset is more challenging with longer length of tail and larger imbalance ratio $r\approx 1000$. The $ISIC-Archive-LT$ dataset is divided into train: val: test = 7:1:2.

All images were resized into 224$\times$224 pixels. We used ResNet-18 and ResNet-50 [7] as our backbones for ISIC-2019-LT and ISIC-Archive-LT respectively. For the SSL pre-training, we used Adam optimizer with a learning rate of 2$\times 10^{-4}$ with a batch size of 16. For the classification task, we used Adam optimizer with a learning rate of 3$\times 10^{-4}$ and a batch size of 256. Some augmentations techniques were applied such as random crop, flip and colour jitter. An early stopping monitor with the patience of 20 was set when there was no further increase on the validation set with a total of 100 epochs.

For our proposed methods, we warmed up the model with plain cross-entropy loss for 30 epochs. The scaling value $s$ for sampling anchor points was set as 0.1. We queried 10% of the samples in the unsampled pool when there was no further improvement with the patience of 10. Then, the optimizer will be re-initialized with a learning rate of 3$\times 10^{-4}$. For a fair comparison study, we kept all basic hyper-parameters the same on all the comparative methods. We reported the average of 5-trial running in all presented results. All experiments are implemented by PyTorch 1.8.1 platform on a single NVIDIA RTX 3090.

The comparative methods selected in this study can be briefly introduced as follow: (1) plain class-balancing re-sampling (RS) and re-weighting (RW) strategies; (2) information augmentation: MixUp [21] and OLTR [18]; (3) Modified re-weighting loss functions: Focal Loss [16], Class-Balancing (CB) Loss  [2] and label-distribution-aware margin (LDAM) Loss [1]; (4) Curriculum learning-based methods: Curriculum Net [6] and EarLy-exiting Framework (ELF) [3].

To investigate what makes our proposed methods performant, we conduct ablation studies and report the results in Table 3. We first evaluate the selection of pre-training methods. It can be found that using SSL pre-training with anchor points can well boost the accuracy of the inference model, where Random denotes the random selection of samples and Edge Points denotes those samples away from the prototypes. Finally, we validated the effect of different querying strategies. Compared with random querying, uncertainty-based mutual information can help introduce samples more efficiently in an instance-aware manner.