CS-AF: A Cost-sensitive Multi-classifier Active Fusion Framework
for Skin Lesion Classification

Fusion approaches have been widely applied in numerous applications, such as skin lesion analysis [5, 6], human activity recognition [7, 8], active authentication [9], facial recognition [10, 11, 12], botnet detection [13, 14, 15] and community detection [16, 17]. According to whether the weights are dynamically/adaptively assigned to each classifier, the multi-classifier fusion approaches are divided into two categories: (i) static fusion, where the weight assigned to each participating classifier will be fixed after its initial assignment, and (ii) active fusion, where the weights are adaptively tuned depending on the characteristics of given samples in the testing phase. Many conventional approaches, such as the bagging [18], boosting [19, 20] and stacking [21], are static fusion. To date, a few methods attempting to conduct active fusion were also proposed [22, 23, 24]. For instance, Ren et al. [22] proposes to use the decision credibility that is evaluated by fuzzy set theory and cloud model, to determine the real-time weight of a base classifier regarding the current sample in the testing phase. META-DES [23] defines five distinct sets of meta-features to measure the level of competence of a classifier for the classification of input samples, and proposes to train a meta-classifier to determine the rank or weight of a base classifier while facing input samples. DES-MI [24] propose an active fusion approach where the weights are determined via emphasizing more on the classifiers that are more capable of classifying examples in the region of underrepresented area among the whole sample distribution. In our work, we propose a novel active fusion approach, that leverages the “reliability” (Section 3.3) and the “individuality ” (Section 3.4) of the base classifiers to assign the weights dynamically and adaptively.

Convolutional neural networks (CNNs) have achieved the state-of-the-art performance [2, 3, 4] in skin lesion analysis since 2016 (i.e., ISIC 2016 Challenge [2]), where nearly all the teams employed CNNs in either feature extraction or classification procedure. Recently, approaches attempting to apply fusion on CNNs to tackle the skin lesion classification, are proposed [25, 26, 5]. For instance, Marchetti et al. [25] presents a fusion of CNNs framework for the the classification of melanomas versus nevi or lentigines, where five fusion approaches were implemented to fuse 25 different CNN classifiers trained on the same dataset of the same problem to make a single decision. Bi et al. [26] proposes another CNNs fusion framework to tackle the classification of melanomas versus seborrheic keratosis versus nevi, where three ResNet classifiers were trained for three different classification problems via fine-tuning pretrained ImageNet CNNs: the original three-class problem and two binary classifiers (i.e., melanoma versus both other lesion classes and seborrheic carcinoma versus both other lesion classes). Perez et al. [5] conducts a comparison study between two fusion strategies for melanoma classification: selecting the classifiers at random (i.e., among 125 models over 9 CNNs architectures), and selecting the classifiers depending on their performance on a validation dataset. To summarize, most of the existing approaches use static fusion for skin lesion analysis. However, as discussed in Section 1, since the skin lesion datasets are usually limited and statistically biased [2, 3, 4], it is necessary to enable active fusion in such problem. To the best of our knowledge, our work is the first to design and apply active fusion approach in skin lesion classification.

A variety of cost-sensitive machine learning approaches have been proposed to tackle the class imbalance issue in pattern classification and learning problems. Mollineda et al. [27], a comprehensive study on the class imbalance issue, divides most of the cost-sensitive machine learning approaches into two categories: the data-level and the algorithmic-level. The data-level approaches usually manipulate the data distribution via over-sampling the samples of the minority classes or under-sampling the samples of the majority classes. For instance, SMOTE [28] is an over-sampling technique proposed to address the over-fitting problem via synthesizing more of the samples of the minority classes. Several variants of the SMOTE approach [29, 30, 31, 32] are also proposed to solve this issue. The algorithmic-level approaches directly re-design the machine learning algorithms to minimize a customizable loss function, that enables the “cost-sensitive” feature, of the classifier on certain dataset (e.g., improving the sensitivity of the classifier towards minority classes). For instance, Zhang et al. [33] proposes an extreme learning machine (ELM) based evolutionary cost-sensitive classification approach, where the cost matrix would be automatically identified given a specific task (i.e., which error cost more). Iranmehr et al. [34] extends the standard loss function of support vector machine (SVM) to consider both the class imbalance (i.e., the cost) and the classification loss. Khan et al. [35] proposes a cost-sensitive deep neural network framework that could automatically learn the “cost-sensitive” feature representations for both the majority and minority classes, where during the training phase, the proposed framework would perform a joint-optimization on the class-dependent costs and the deep neural network parameters. In this work, we enable the “cost-sensitive” feature in the process of multi-classifier fusion, and employ it in the skin lesion classification problem.

In multi-classifier fusion, we define a classification space, as shown in Figure 1, where there are $m$ classes and $k$ classifiers. Let $\mathcal{M}=\{M_{1},M_{2},\dots,M_{k}\}$ denote the set of base classifiers and $\mathcal{C}=\{C_{1},C_{2},\dots,C_{m}\}$ denote the set of classes. Let $p^{m}_{kj}$ denote the posterior probability of given sample $j$ identified by classifier $M_{k}$ as belonging to class $C_{m}$, where $P_{kj}=\{p^{1}_{kj},p^{2}_{kj},\dots,p^{m}_{kj}\}$ and $\sum_{l=1}^{m}p^{l}_{kj}=1$. Hence, all the posterior probabilities form a $k\times m$ decision matrix as follows:

Since the importance of different classifiers might be different, we assign a wight $w_{i}$ to the decision vector (i.e., posterior probabilities vector) of each classifier $C_{i}$, where $i\in\{1,2,\dots,k\}$. Let $P_{m}(j)$ denote the sum of the posterior probabilities, that sample $j$ belonging to class $m$, of all the classifiers. Then, we have

The final decision (i.e., class) $D(j)$ of sample $j$ is determined by the maximum posterior probabilities sum:

Conventional multi-classifier fusion approaches either use the same weight for all the classifiers (i.e., average fusion) or use static weights that will not be changed after its initial assignment during the training phase. As illustrated in Figure 1, our weights (i.e., $w_{k}=\frac{O_{k}+S_{k}}{2}$) contains two components: (i) the objective weight $O_{k}$ that is static and determined by the prior knowledge obtained through the training phase (Section 3.3), and (ii) the subjective weight $S_{k}$ that is dynamic and calculated by the posterior knowledge obtained through the testing phase (Section 3.4). To be simplified, we assign the same weight, i.e., $0.5$, on both $O_{k}$ and $S_{k}$, while combining them together.

As discussed in Section 1, given a sample, different outputs (i.e., the correct prediction or all kinds of errors) of a classifier should result in different costs. For instance, misclassifying a more severe lesion to a benign or less severe lesion should have relative higher cost. Let $c_{pq}$ denote the cost of classifying an instance belonging to class $p$ into class $q$. Then, we obtain a cost matrix as follows:

Let $W=\{w_{1},w_{2},\dots,w_{k}\}$ be a fusion weight vector, and $\mathcal{W}$ be the fusion weight vector space, where $W\in\mathcal{W}$. The goal of cost-sensitive multi-classifier fusion is to find the $W^{*}\in\mathcal{W}$, that can minimize the average cost of the fusion approach’s outcomes over all the testing samples. In Section 4.3, we provide certain principles to design a good cost matrix. We also provide two examples of cost matrices, that emphasize on different skin lesion classes, based on our literature references and those principles.

The objective weights are designed according to the classifiers’ reliability to recognize the particular skin lesions, which is determined by the prior knowledge obtained through the training phase. In the training phase, we separate all the labelled data into two parts: training data and validation data. As shown in Figure 2, computing the objective weights in our framework contains three steps:

$\bullet$ Classifier build. We prepare a set of base classifiers, where all the classifiers might have different CNN architectures, use different size of the training data, or use different subset or classes distributions of the training data. In this step, we trained 96 base classifiers, more details are introduced in Section 4.2.

$\bullet$ Reliability validation. Let $r_{i}$ denote the reliability of a base classifier $M_{i}$, that is designed to describe the average recognition performance of the classifier on the validation data. Higher accuracy and less error on the validation data usually means higher reliability. Hence, we use the confusion matrix result of each base classifier on the same validation dataset as its reliability, where a confusion matrix [36] is a table that is often used to describe the performance of a classifier on a set of validation data for which the true values are known. It allows easy identification of confusion between classes, e.g., one class is commonly mislabeled as the other. Many performance measures could be computed from the confusion matrix (e.g., F-scores). As such, we use $r_{i}^{pq}$ to denote the probability of a base classifier $M_{i}$ classifying an instance belonging to class p into class q.

$\bullet$ Cost-sensitive adjustment. As described in Section 3.2, we would like to enable the “cost-sensitive” feature in the design of our objective weights. As shown in Figure 3, for each classifier $M_{i}$, we use an element-wise multiplication between its reliability $r_{i}$ (confusion matrix) and the customized cost matrix (Section 3.2) to formulate a cost-sensitive confusion matrix, where all the results/errors in the confusion matrix have been adjusted based on the cost matrix. Then, we use the micro-average F1-score [37] of the cost-sensitive confusion matrix to define the objective weight of each base classifier, and all the object weights are automatically normalized to $(0,1]$.

The subjective weights are designed according to the relative confidence of the classifiers while recognizing a specific previously “unseen” image (i.e., individuality), which are calculated by the posterior knowledge obtained through the testing phase. As shown in Figure 4, computing the subjective weights in our framework contains three steps:

$\bullet$ Sample evaluating/testing. Each sample is evaluated/tested through all the classifiers to obtain the corresponding decision vectors (i.e., the soft labels).

$\bullet$ Individuality calculation. We consider the individuality of a classifier as its relative class discriminative power regarding a given individual sample. A classifier can easily identify the class of a given sample in the testing phase, if its posterior probabilities of the corresponding decision vector is highly concentrated in one class, and the misclassification rate would also be low. On the contrary, if the distribution of the posterior probabilities is close to uniform, the classifier shows its difficulty in discriminating the class of the given sample. Also, different classifiers would present different distribution of the posterior probabilities in the decision vectors while testing the same sample. Hence, we define the the individuality $i_{k}$ of a classifier $M_{k}$ using the posterior probabilities distribution as follows:

where $p^{*}_{kj}$ is the largest posterior probability value in $P_{kj}$.

$\bullet$ Normalization. Since the subjective weights are relative values among all the base classifiers, we normalize each individuality $i_{k}$ to the subjective weight $S_{k}\in[0,1]$ as follows:

where $i^{min}=\underset{j\in{1,2,\dots,k}}{min}i_{j}$ and $i^{max}=\underset{j\in{1,2,\dots,k}}{max}i_{j}$.

In our experiments, we utilized the well known ISIC Challenge 2019 dataset [38, 39, 3]. Since the ground truth of the original testing data was not available, we only employed the original training data without meta-data in our experimental evaluation. This dataset (i.e., the original training data of the ISIC Challenge 2019) contains 25,331 images in total, coming from 3 source datasets: BCN_20000 [38], HAM10000 [39] and MSK [3]. It depicts 8 skin lesion diseases (i.e., 8 classes): melanoma (MEL, 4,522 images), melanocytic nevus (NV, 12,875 images), basal cell carcinoma (BCC, 3,323 images), actinic keratosis (AK, 867 images), benign keratosis (BKL, 2624 images), dermatofibroma (DF, 239 images), vascular lesion (VASC, 253 images) and squamous cell carcinoma (SCC, 628 images). We split the entire 25,331 images into training (80%), validation (5%) and testing (15%) datasets.

To evaluate the performance of our proposed CS-AF framework using the base classifiers that are trained from the datasets with different classes distributions, we designed 4 training datasets that have different classes distributions. For instance, one training dataset could have balanced classes distribution, and the other training datasets could have unbalanced classes distributions in different ways. The details (i.e., classes distributions) of each training dataset are shown in Table 1 and described as below:

• 1.

  Dist-1: This training dataset follows the classes distribution of the original training dataset of the ISIC Challenge 2019 dataset.

• 2.

  Dist-2: This training dataset contains evenly distributed number of samples for all classes.

• 3.

  Dist-3: This training dataset contains more samples for cancerous lesion classes, and less samples for benign lesion classes.

• 4.

  Dist-4: This training dataset contains less samples for cancerous lesion classes, and more samples for benign lesion classes (i.e., the opposite order of classes distributions as in Dist-3).

To generate different training datasets satisfying different classes distributions described above, we utilized data augmentation techniques to generate more images for the skin lesion classes lacking of images, and randomly sampled smaller portions from the classes with superfluous images. The main data augmentation techniques utilized are rotation (for 45, 90, 135, 180, 225, 270 and 315 degrees, respectively), horizontal flip or the combination of both. We also utilized the same strategy to generate the validation and testing datasets, to ensure the numbers of samples of all classes are equal, where there are approximate 200 samples of each class in the validation dataset, and approximate 500 samples of each class in the testing dataset.

In addition, to evaluate the performance of our proposed CS-AF framework using the base classifiers that are trained from different subsets of the training dataset, for each of those four training datasets that have different classes distributions, we randomly select 70% of it to produce another sub-dataset, namely, Sub-70. Therefore, in our experimental evaluation, there are 8 different training datasets in total (i.e., Dist-1, Dist-1 Sub-70, Dist-2, Dist-2 Sub-70, Dist-3, Dist-3 Sub-70, Dist-4 and Dist-4 Sub-70).

We prepared the base classifiers to evaluate the fusion approaches performance using 12 different popular CNN architectures (as shown in Table 2) with parameters pre-trained on ImageNet [50]. Different CNN architectures have different size of input images:

• 1.

  331$\times$331: PNASNet-5-Large, NASNet-A-Large.

• 2.

  320$\times$320: ResNeXt101-32$\times$16d.

• 3.

  299$\times$299: Xception, Inception-V4, Inception-V3, InceptionResNet-V2.

• 4.

  224$\times$224: SENet154, Dual Path Net-107, SE-ResneXt101-32$\times$4d, EfficientNet-b7, ResNet152.

All the base classifiers were fine-tuned by stochastic gradient decent (SGD) with learning rate $10^{-3}$ and momentum 0.9. The learning rate degraded in 20 epochs by 0.1. We stopped the training process either after 40 epochs or while the validation accuracy was failed to improve for 7 consecutive epochs. Our experiments were implemented using Pytorch, running on a server with 4 GTX 1080Ti 11 GB GPUs. To keep the same batch size 32 in each evaluation, and due to the memory constraint of single GPU, certain CNN architectures were trained with more GPUs:

• 1.

  2 GPUs: SENet154, EfficientNet-b7, Dual Path Net-107.

• 2.

  4 GPUs: PNASNet-5-Large, ResNeXt101-32$\times$16d, NASNet-A-Large.

Table 2 illustrates the performance (i.e., accuracy) of all 96 base classifiers on the testing dataset.

As discussed in Section 3.2, it is necessary to have an appropriate cost matrix to enable our “cost-sensitive” feature in our active fusion framework. Hence, we create several principles to design an adequate cost matrix:

• 1.

  All the costs should be positive, since it will be item-wise multiplied with the confusion matrix, and we don’t want to get non-positive values in our cost-sensitive confusion matrix.

• 2.

  The cost of the correct predictions should depend on the relative severeness of the corresponding disease. For instance, it should be more valuable (i.e., less cost) to classify a more severe disease (i.e., melanoma) correctly. To figure out the relative severeness relationships among all eight skin lesion classes and better design our cost matrix, we referred to some literature articles regarding to the severeness of those diseases [51, 52, 53, 54, 55, 56, 57, 58, 59]. To be simplified and enable the evaluation of our work, based on those references, we heuristically ordered the severeness of the 8 skin lesion classes (from the most severe to the least one) as follows: melanoma (MEL), squamous cell carcinoma (SCC), basal cell carcinoma (BCC), melanocytic nevus (NV), actinic keratosis (AK), dermatofibroma (DF), vascular lesion (VASC), benign keratosis (BKL).

• 3.

  The relative costs of different incorrect predictions should be based on their relative severeness. For instance, misclassifying melanoma (i.e, a deadly cancerous lesion) as benign keratosis should result in much more cost than the opposite scenario.

• 4.

  The cost of correct predictions should be no more than the cost of incorrect predictions.

As such, we designed two examples of cost matrices that emphasize on different skin lesion classes (i.e., cancerous lesions vs. benign lesions) to evaluate our work, as illustrated in Figure 5a (i.e., Cost Matrix A) and Figure 5b (i.e., Cost Matrix B).

Let us take the design of Cost Matrix A, that emphasizes on cancerous lesions (i.e., the cost of misclassifying a cancerous lesion is much more than a benign lesion), as an example. Firstly, we assign the cost of correct prediction of each skin lesion class, i.e., $c_{ii}$, $i=1,2,\dots,m$ (as defined in Section 3.2), according to the relative disease severeness, where predicting a more severe skin lesion class correctly should result in less cost. For instance, we set the cost of correct prediction of MEL (i.e., the most severe one) as 1, and the cost of correct prediction of BKL (i.e., the least severe one) as 8. Secondly, to calculate the relative cost of each incorrect prediction, we follow the equation below:

where as defined in Section 3.2, $c_{ij}$ denote the cost of classifying an instance belonging to class $i$ into class $j$. For instance, if the cost of correct prediction of MEL is 1 and the cost of correct prediction of BKL is 8, the cost of misclassifying an instance belonging to MEL into BKL would be $(\frac{8}{1})^{2}=64$. Last but not least, to ensure the cost of correct predictions are always no more than the cost of incorrect predictions, without loss of generality, we normalized the costs of misclassifications to integers between 16 and 200, using min-max scaling. Figure 5a shows the final result of our designed Cost Matrix A.

To evaluate our framework under different cost matrices, we also designed a Cost Matrix B (as shown in Figure 5b), that emphasizes on benign lesions (i.e., the cost of misclassifying a benign lesion is much more than a cancerous lesion). Cost Matrix B follows the same design steps as Cost Matrix A, other than considering an exactly reverse order of the severeness. For instance, in the design of Cost Matrix B, melanoma became the “least severe” one and benign keratosis became the “most severe” one.

Our two examples of cost matrices are just for our experimental evaluation purposes. To utilize our framework, the other researchers could always create the cost matrix with their own demands or based on their own best domain expert knowledge.

As described in Section 4.2, we have prepared 96 base classifiers. To evaluate the effectiveness of our active fusion approach extensively, we use different subsets of the 96 base classifiers to form a fusion model. Specifically, given a fusion approach, each time, we randomly select $N$ classifiers among all 96 base classifiers to form a fusion model and evaluate its performance, where $N=8,16,24,32,40,48,56,64,72,80,88,96$. For each specific $N$, we will repeat the random selection experiments for 100 times, and use the averaged performance as the final results.

To demonstrate the effectiveness of our approach, a comparison between two CS-AF implementations (using two different cost matrices to compute the objective weights) and three other fusion approaches has been conducted. The contestants are:

• 1.

  Max Voting Fusion is a static approach, where predictions are combined from multiple base classifiers and only the predicted class with the highest votes will be included in the final prediction.

• 2.

  Average Fusion is another static approach, where it averages the decision vectors of multiple base classifiers and uses the averaged decision vector to make the final prediction.

• 3.

  AF is a baseline active fusion approach by removing the cost-sensitive adjustment step from CS-AF while calculating the objective weights.

• 4.

  CS-AF (Cost Matrix A) is our approach while computing its objective weights using Cost Matrix A.

• 5.

  CS-AF (Cost Matrix B) is our approach while computing its objective weights using Cost Matrix B.

Given a contestant fusion approach, we evaluate its effectiveness in terms of (i) its averaged accuracy on our testing dataset (as shown in Figure 6a), (ii) its total cost on our testing dataset specified by Cost Matrix A (as shown in Figure 6b), and (iii) its total cost on our testing dataset specified by Cost Matrix B (as shown in Figure 6c). The total cost could be calculated by the sum of the item-wise product between the confusion matrix resulted from our testing dataset and the specified cost matrix. By definition, better fusion approach usually leads to higher accuracy on the testing dataset and lower total cost specified by certain cost matrix. From the results (Figure 6), we could observe that:

• 1.

  Compared with the best performed base classifier, ResNeXt101-32$\times$16d, as shown in Table 2, our two implementations of CS-AF and AF consistently achieve over 2%-5% higher accuracy on the testing dataset.

• 2.

  For all the fusion approaches, as more base classifiers involved, the accuracy tends to increase and the total cost tends to decrease.

• 3.

  In terms of the accuracy, our two implementations of CS-AF and AF consistently outperform the static fusion approaches (Max Voting and Average), and CS-AF (Cost Matrix B) consistently achieves the best accuracy. Hence, it presents that our proposed active fusion approach is rather effective towards enhancing the overall accuracy.

• 4.

  In terms of the total cost, CS-AF consistently outperforms the other fusion approaches (Max Voting, Average and AF). For instance, while calculating the total cost using Cost Matrix A, CS-AF (Cost Matrix A) always achieves the lowest total cost, and while calculating the total cost using Cost Matrix B, CS-AF (Cost Matrix B) always obtains the lowest total cost. Thus, it demonstrates that our proposed cost-sensitive active fusion approach could adapt to different cost matrices and is optimized to achieve the lowest total cost.

As discussed above, our proposed CS-AF could adapt to different cost matrices and is optimized to achieve the lowest total cost under a specified cost matrix, namely “cost-sensitive”. In this section, we would like to analyze how such “cost-sensitive”, while using certain cost matrix, influence the performance of CS-AF on certain single skin lesion classes, thus reducing the total cost. We evaluate single class performances using sensitivity and specificity, defined as follows:

where $TP$ denotes the number of true positives and $FN$ denotes the number of false negatives.

where $TN$ denotes the number of true negatives and $FP$ denotes the number of false positives.

Figure 7 and Figure 9 illustrate the sensitivity and specificity results of each single class of CS-AF (Cost Matrix A) and CS-AF (Cost Matrix B), respectively. We could observe that

• 1.

  Compared with CS-AF (Cost Matrix B), CS-AF (Cost Matrix A) tends to achieve higher sensitivity on more severe cancerous lesion classes (i.e., melanoma, squamous cell carcinoma and basal cell carcinoma), and lower sensitivity on less severe benign lesion classes (i.e., benign keratosis, vascular lesion, dermatofibroma, actinic keratosis and melanocytic nevus)

• 2.

  Compared with CS-AF (Cost Matrix A), CS-AF (Cost Matrix B) tends to achieve higher specificity on those more severe cancerous lesion classes, and lower specificity on those less severe benign lesion classes.

As described in Section 4.3, Cost Matrix A emphasizes on cancerous lesions (i.e., the cost of misclassifyinga cancerous lesion is much more than a benign lesion), while Cost Matrix B emphasizes on benign lesions (i.e., the cost of misclassifyinga benign lesion is much more than a cancerous lesion). While using Cost Matrix A to compute the objective weights of our CS-AF implementation (i.e., CS-AF (Cost Matrix A)), it tends to increase the TP and FP of cancerous lesion classes and decrease the FN and TN of benign lesion classes, thus resulting in higher sensitivity and lower specificity for cancerous lesion classes. CS-AF (Cost Matrix B) also works in such way accordingly. Therefore, the observations demonstrate that our proposed CS-AF is “cost-sensitive”, where its performance on certain single skin lesion classes could be flexibly influenced by or adapted to certain well designed cost matrices.