Self-Supervised Multi-Modality Learning for Multi-Label Skin Lesion Classification

Skin lesion classification has seen significant advancements with the introduction of CNNs [22] which has become the preferred technology for developing CAD systems [23, 24, 25]. In skin lesion classification, CNNs have shown superior performance compared to traditional methods based on handcrafted features [12, 26]. Researchers have extended this approach by using more advanced techniques, such as deep CNNs with more layers [8], attention learning mechanism [9], and regularization strategies for small and unbalanced datasets [27]. However, these methods often overlooked clinical images, which are crucial for precise decision making. To address this limitation, Ge et al.[10] proposed a multi-modality learning method that utilized both dermoscopy and clinical images by applying separate CNNs for each modality. Subsequent research has built on this approach by incorporating multi-scale feature fusion modules [3] and adversarial learning with attention mechanisms [28] to capture both correlated and complementary information from two image modalities. In addition, researchers have explored the detection of dermoscopic attributes (multi-label classification) from the seven-point checklist to improve the classification performance [1]. For example, Fu et al.[11] proposed a graph-based model to capture the interrelationships between different labels. Similarly, Tang et al. [29] developed a two-stage learning scheme, where dermoscopy and clinical image features were integrated in the first stage, which were then integrated with patient metadata in the second stage to capture correlations between labels.

SSL has emerged as a promising alternative to alleviate the problem of expensive and time-consuming annotation processes [30]. This is particularly relevant in the context of medical imaging, where annotated data are scarce due to the complicated data acquisition procedures [31]. One of common SSL approaches is to use Contrastive Learning [32] which uses a pretext task that maximizes similarities between similar data instances while minimizing them with dissimilar instances. For example, it maximizes the similarities between augmented views (e.g., rotated, masked views) of the same image and minimize similarities with augmented views of different images [16, 17, 18, 19]. For example, Azizi et al. [31] trained a model on an unlabeled dataset and used a pretext task based on multiple images of the same clinical case to improve skin lesion classification. Recently, there was a systematic review that evaluated the use of various SSL algorithms for skin lesion classification [33]. There were also other studies of using SSL to address common skin lesion classification challenges, e.g., long-tail out-of-distribution problem [34] and light weight models [35]. However, all these SSL-based methods focused on using a single medical imaging modality and cannot be directly applied to multi-modality learning. While multi-modality learning can be implemented through the simple concatenation of multi-modality image features, this is not optimal to derive complementary information in a SSL setting. As also pointed by Li et al. [36], naïve concatenation is not an efficient way and could heavily decrease the model performance due to the domain differences between different imaging modalities. To address this issue, Zhang et al. [37] proposed to use the Mean Squared Error (MSE) to align multi-modal feature maps and designed a new contrastive loss to enforce the network to focus on the similarities of segmentation masks from paired modalities as well as dissimilarities of unpaired multi-modal data. Huang et al. [38] proposed a SSL algorithm for four-modality ultrasound learning, where Mean Absolute Error across different modalities was minimized to ensure that high-level image features extracted from different modalities can be similar.

MLC considers a scenario where a set of class labels is assigned to a single data instance. In this context, prior research has primarily concentrated on devising models for understanding relationships between labels [39], including approaches such as one-vs-all classifiers [40], tree structures [41], and graph structures [42]. Medical imaging field also has adopted MLC, as exemplified by Guan et al.’s use of a residual attention learning framework for chest X-ray image classification. It assigned different weights to different spatial regions based on multi-labels [43]. Similarly, Liu et al. [44] enhanced model robustness by maintaining the consistency of relationships among different samples under perturbations. Zhang et al. [45] employed a triplet attention network with a Transformer to make use of multi-labels together with spatial and category attention features.

We used a publicly available multi-modality and multi-label skin lesion dataset (Derm7pt) [1] in our experiments. It contains a total of 1,011 studies. The dataset is divided into 413 studies for training, 203 studies for validation, and 395 studies for testing, according to [1]. Each study contains a pair of dermoscopy and clinical images, a diagnosis (DIAG) label, and seven-point checklist labels. The DIAG label consists of 5 types of skin lesions including Basal Cell Carcinoma (BCC), Nevus (NEV), Melanoma (MEL), Miscellaneous (MISC), and Seborrheic Keratosis (SK). The seven-point checklist labels contain Pigment Network (PN), Blue Whitish Veil (BWV), Vascular Structures (VS), Pigmentation (PIG), Streaks (STR), Dots and Globules (DaG), and Regression Structures (RS). Each seven-point checklist label has different number of classes including Absent (ABS), Typical (TYP), Atypical (ATP), Present (PRS), Regular (REG), and Irregular (IR). The size of dermoscopy images varies from 474 × 512 to 532 × 768 pixels and the clinical images vary from 480 × 512 to 532 × 768 pixels.

In this work, we used a SSL method SimCLR [16] as the base pre-training strategy. The workflow of SimCLR is shown in Fig. 2. Firstly, two separate data augmentation sets $\{\mathcal{T},\mathcal{T}^{\prime}\}$ from the same family of augmentations (including random resized cropping, color jitter, random horizontal flip, and random Gaussian blur) randomly transform any given image sample $x$ into two augmented views $x^{1}$ and $x^{2}$, which are considered as a positive pair. Then, an encoder network $f(\cdot)$ extracts image features $h^{1},h^{2}$ from augmented views, respectively. The choice of encoder network is flexible and can be any CNN architecture. Afterward, a projection head $g(\cdot)$ maps learned image representations into a latent space $z$. We used the Multi-Layer Perceptron (MLP) as our projection head. Lastly, the contrastive loss is applied to the latent space $z$, aiming to maximize the similarities between positive pairs. The loss function for image $x_{i}$ is defined as:

where $N$ denotes the batch size, $\tau$ is a temperature hyperparameter. $1_{\left[i\neq j\right]}\in\left\{0,\ 1\right\}$ is an indicator function equaling to 1 if and only if $i\neq j$. $e\left(\cdot\right)$ is the exponentiation operation, and $\sigma\left(u,v\right)=u^{\top}v/||u||\,||v||$ denotes the cosine similarity function.

The overview of our method is shown in Fig. 3. Initially, modality-specific features from dermoscopic and clinical images were extracted using SimCLR. Subsequently, we pre-trained the multi-modality models by maximizing similarities between paired multi-modality images of the same patient. Following this, the extracted image features were projected into distinct label-specific embedding spaces. A label-relation-aware module was then used to learn correlation between labels. Lastly, we channeled the outputs into clusters, generating pseudo-multi-labels for SSL multi-label pre-training.

Given pairs of dermoscopy and clinical images, we utilized separate CNNs, named dermoscopy branch and clinical branch, to extract corresponding image modality features. These two branches have identical architecture but independent weight updates, which helps to optimize each branch for different image modalities. To enable efficient multi-modality representation learning, three pretext tasks are defined. The first and second pretext task are to apply SimCLR in each model branch to extract specific features from corresponding modalities. We defined the loss functions using Equation 1 as follows:

where $z$ in $\mathcal{L}_{derm}$ comes from dermoscopy images, whereas $z$ in $\mathcal{L}_{clinic}$ is derived from clinical images. We used these loss functions to solve the first and second pretext tasks. In addition, we propose a third task that jointly utilizes both dermoscopy and clinical images, allowing complementary representation learning of the two modalities for multi-modal fusion. The intuition behind our design is that pairs of multi-modality images have more similarities than others, i.e., dermoscopy and clinical images of the same case have maximum mutual information under different augmented views. We implemented this idea by 1) introducing two extra projection heads to map extracted dermoscopy and clinical features into a shared embedding space and 2) maximizing the agreement between randomly augmented views of the same case but different modality data sample by the modified contrastive loss:

where $z$ is computed from dermoscopy images while $z^{\prime}$ is calculated from clinical images. We applied these three tasks to train the model by adopting multi-task learning and defined the final loss function as:

Naïve solution. Since multiple label predictions are derived from the single image representation and the number of classes is different for each label, we adopted separate classifiers for every label prediction. The classifier $h\left(\cdot\right)$ was built by a label projection head $p\left(\cdot\right)$ and a classification head $q\left(\cdot\right)$ such that $h\left(\cdot\right)=q\left(p\left(\cdot\right)\right)$. Here, $p\left(\cdot\right)$ consists of an MLP aiming to filter label-specific features and $q\left(\cdot\right)$ is a single fully-connected (FC) layer to make final predictions. To enable self-supervised learning of the multi-label classifier, we used the k-means clustering algorithm [46] to generate pseudo-multi-labels. We iterated the clustering process independently for $K$ times to produce multiple labels for the same data point where $K$ equals to the number of unique labels in the dataset. We then utilized these generated pseudo-multi-labels to update the parameters of the classifier by optimizing the cross-entropy loss function which is defined as follows:

where $x_{i}$ is the $i_{th}$ image and $y_{i}$ is the corresponding pseudo-multi-labels containing $\{y_{i,1},\ldots,y_{i,K}\}$. $h_{k}\left(\cdot\right)$ denotes the $k_{th}$ classifier.

Label-relation-aware solution. The above naïve solution, however, yielded degraded results in our preliminary experiments (Section V.B.2), which overlooked the relationships between labels. We therefore further refined each label embedding by understanding the relationships between other label embeddings guided by an attention mechanism [47]. Attention mechanism has been successfully used to capture relationships between feature representations. Formally, we inserted a function $W\left(\cdot\right)$ before $q\left(\cdot\right)$ and fed outputs of all $p\left(\cdot\right)$ into it. Then, we rewrote the classifier function as:

where $\{p_{1},\ldots,p_{K}\}$ are label projection heads for $K$ unique labels. Here, the features specific to each label only contain information about that label. However, our label-relation-aware module come into play to connect all labels information and learn the relationships between them. As a result, the prediction of a single label involves contributions from other label information. In this work, we used $W\left(\cdot\right)$ as the encoder layer of Transformer [47].

We adopted a vanilla implementation of multi-modality and multi-label classification scheme (as shown in Fig. 4) to emphasize the effectiveness of our proposed SSL pre-training method. Following the design of multi-modality model fusion in preliminary work EmbeddingNet [48], we applied two separate branches to extract dermoscopic and clinical features. These two modality features were then concatenated and fed into a subsequent classifier to make the final predictions. This model was referred as the Baseline in the Section IV. Experiments, and we initialized both the branches and the classifier using our SM3 pre-trained weights.

We used a deep learning library PyTorch [49] to implement our algorithm. All the experiments were conducted on two NVIDIA RTX 3080Ti 12GB GPUs. For fair comparisons, we used ResNet-50 [25] as the CNN backbone since ResNet-50 has been commonly applied for various skin lesion classification and segmentation tasks including [3, 28, 11, 29]. The output dimension of the projectors in contrastive learning was set to 128. For multi-label SSL, we set the label projection head as a single-layer MLP with a dimension of 512. We used a single Transformer encoder layer with a head of 1, a feed forward dimension of 128 and a dropout rate of 0.1 for our label-relation-aware module . All these hyperparameters were chosen based on our empirical studies. We resized all of images into 224 × 224. Code is available at https://github.com/Dylan-H-Wang/skin-sm3.

The primary results of the experiment are presented in Table I. We first evaluated the effectiveness of the proposed SM3 representations via linear probing (SSL + Linear Probing). When compared to SimCLR, our SM3 showed consistent improvements including a 2.2% increase in mean AUC, 3.8% increase in mean Sens, 2.1% increase in mean Spec, and 3.6% in mean Prec, although SimCLR performed better in some categories, e.g., RS category. SSD-KD, based on SSL and knowledge distillation, achieved a second-best performance with an AUC of 79.6, mean Sens of 9.8, mean Spec of 96.7, and mean Prec of 31.7. In comparison, our SM3 had much higher mean Sens (+25.1%), mean Prec (+29.8%) and relatively higher mean AUC (+0.8%) but lower mean Sens (-3.1%). To understand the upper bound of our SSL linear probing, we compared the SM3-initialized Baseline model with the supervised ImageNet-initialized Baseline model, with SM3 resulting in a higher mean Specificity (+2.7%), a small gap in terms of mean AUC (-0.9%), mean Sensitivity (-7.6%), and mean Precision (-5.2%).

We conducted experiments to assess the effectiveness of SM3 in improving various backbones (SSL + Fine-tuning). After fine-tuning the SM3-initialized Baseline model, the mean AUC improved from 81.3 to 82.9, surpassing both Inception-Combined (mean AUC of 81.5) and HcCNN (mean AUC of 82.5). This improvement was consistent regarding mean Spec and mean Prec. We also experimented with another existing method, F4M-FS by replacing the ImageNet-pre-train weights with our SM3-pre-trained weights while keeping other components unchanged. We found that the SM3-initialized FM4-FS achieved 1% increase in mean AUC, 4.2% in mean Spec and 12.6% in mean Prec. Compared to the current supervised state-of-the-art AMFAM which obtained a mean AUC of 84.5, mean Sens of 57.9, mean Spec of 86.7, and mean Prec of 53.5, SM3-initialized FM4-FS outperformed it by 0.1% in mean AUC, 6.6% in mean Spec and 22.5% in mean Prec but with 17.3% lower in mean Sens.

As shown in Table I, most methods performed relatively well due to the use of complex multi-modality fusion techniques, such as class-balanced sampling and multi-task loss in Inception-Combined [1], concatenation of intermediary image features in HcCNN [3], adversarial fusion with attention mechanism in AMFAM [28], and hierarchy fusion at the feature and decision levels in F4M-FS [29]. However, most of these methods ignored the MLC setting and did not exploit the interrelationships among labels. GIIN [11], on the other hand, additionally leveraged a graph module to model the label relationships which resulted in improved performance. Compared to Baseline-50-ImageNet, our SM3-fine-tuned method achieved a 1.6% increase in mean AUC, surpassing established multi-modal fusion strategies like Inception-Combined and HcCNN. Similarly, our SM3 increased the recently published F4M-FS performance with an improved mean AUC of 1%. This indicates that SM3-pre-trained weights could be an alternative to ImageNet-pre-trained weights for improving the generalizability of different methods in skin lesion analysis.

In linear probing experiments, our method demonstrated a mean AUC enhancement of 2.2% when compared to the established SimCLR [16]. Furthermore, in comparison to the recent SSL skin lesion classification approach of SSD-KD [35], SM3 exhibited a mean AUC increase of 0.8%. It is worth noting that the degradation in performance with mean Sens metric is expected. This is mainly attributed to the fact that a large proportion of contributions are coming from the VS category, which tends to have low Sens with a relatively high Spec. As indicated by Kawahara et al. [1], the low sensitivity and high specificity is likely attributed to the substantial class imbalance issue in the dataset, where there only exists 71 IR studies in the VS category out of 1,011 studies. By pre-training the model on a skewed dataset without labels, this discrepancy was accumulated resulting in a lower Sens score with a higher Spec score. Nevertheless, we identified that SM3 was effective in dealing with imbalanced data in the seven-point dataset when compared to the other two SSL methods. For instance, IR in the VS category has extremely limited number of cases when compared to the more prevalent REG, and both compared methods struggled to correctly classify IR. In contrast, our method leveraged multi-modality and multi-label techniques managing to correctly classify this minority class. These results underscore the efficacy of our multi-modality SSL design, which harnesses the supplementary potential of dermoscopic and clinical image features to extract enhanced discriminative skin attributes. Moreover, the inclusion of self-supervised multi-label pre-training augments the model’s capacity to glean intricate interrelationships among labels, thereby contributing to a more consistent and reliable classification performance.

We found that naïve SSL pre-training (i.e., SimCLR strategy) with multi-modal data contributed to improving the baseline performance when compared to the commonly used ImageNet-pre-trained weights. This finding is consistent with previous work by Menegola et al [15], where the domain gap between natural images and skin lesion images degraded performance. The SimCLR strategy, however, is not optimal for multi-modality learning since the mutual information between two modalities are not leveraged during the pre-training. Nevertheless, for the multi-modal SSL, inappropriate multi-modal fusion (i.e., concat) could hinder the learning of meaningful representations. This observation is consistent with findings from a previous work [36] that naïve concatenation may result in a worse performance due to domain shift among different modalities. For example, the strategy sep_shared was better than concat but still inferior to the SimCLR strategy in terms of the mean AUC. In contrast, the sep_sep strategy demonstrated higher performance overall and we attribute this to the application of separate projection heads which can focus on independent image modality features and decide how to map them to increase their similarities. Furthermore, we also identified that contrasting naïve concatenation (concat) did not contribute to the learning of mutual information between the two modalities, and directly contrasting two modality features (sep_shared and sep_sep) was more effective giving better pair matching performance. It is noteworthy that although sep_shared was capable of learning more mutual information than concat and SimCLR strategies, its classification accuracy was lower than that of SimCLR. This suggests that an inefficient multi-modality pre-training, i.e., sharing the same projection head, learns trivial complementary multi-modality information and hinders the extraction of individual modality features.

In addition, we observed that directly pre-training a multi-label classifier on image features without label projection heads (no_proj) was not helpful, and simply projecting image features into label embeddings (proj) can disrupt self-supervised multi-label learning. We hypothesize that such failure was caused by the independent learning of label projection heads. A naïve label correlation learner (msa) had trivial contributions for multi-label SSL, whereas complex model (te) cannot achieve reasonable results neither attributing to the fact that the performance of the Transformer based architecture is heavily reliant on the use of large training dataset, which cannot be satisfied with the current experimental dataset. Therefore, it is essential to define an optimal learning strategy, i.e., the Transformer encoder layer, for learning the correlations among labels.

Pre-training on an imbalanced dataset, without labels, can accumulate to discrepancies, such that it results in a more severe class imbalance issue. To mitigate this, we will consider developing pre-training models with consideration of imbalanced dataset by combining predictions from multiple models trained on different subsets of the dataset and focusing on minority class samples. Although our method was demonstrated for skin lesion dataset, we suggest that it is applicable to other multi-modality and multi-label imaging datasets where it retains informative mutual features between modalities and relationships between labels, such as with classification on multi-modality PET-CT and PET-MR datasets which contains two imaging modalities with complimentary information.