A Multimodal Approach to The Detection and Classification of Skin Diseases

Large datasets comprised of thousands of annotated images are collected from diverse sources, including medical archives, research repositories, and clinical studies. Three such skin disease datasets are available on Kaggle (17, 18, 19). The first dataset consists of 27,153 images classified into ten classes of skin diseases and the second dataset includes 19,600 images classified into 23 types of skin diseases/classes. The first dataset has good distributions of the 10 classes and was used by Hammad (7), who focused on only two classes, which are 1,677 images for eczema and 2,055 images for psoriasis. The third dataset (19), derived from the ISIC competition source website, contains about 23,618 images classified into 8 types of skin diseases. This study combines the three datasets; however, some images are duplicates or not related to skin diseases. The images were manually checked to see whether they are relevant. Afterwards, hashing was done to remove any duplicate images. The final dataset comprised of 36,995 images across 26 types of skin disease. Unfortunately, class distribution is not perfect because of the large variation in image numbers for different diseases. Some diseases only have around 200 images while others have over 7,900 images. This highlights one of the challenges of getting skin disease images for AI applications which our study overcomes. Table 1 shows the classes and number of images in each dataset used in this study. The images were divided into 80% for training and 20% for validation/testing.

In addition to images of skin diseases, symptoms of these diseases are another source of information for classifying what ailment a patient has. This study will add those disease symptoms to help skin disease diagnosis. To do so, the symptoms of skin disease were summarized and aggregated and then turned into patient stories. The symptoms of each skin disease were obtained through Google. Table 2 shows examples of three diseases and their symptoms summarized from Google. Since the symptoms of each skin disease cannot be directly input into an LLM, this study uses ChatGPT to generate a story to describe a patient’s symptoms based on select symptoms of the disease. The final dataset consists of 10 stories per skin disease with a total of 260 text data. Table 3 shows an example of 10 stories for the eczema skin disease.

Doctors can discern most skin diseases based solely on images as they generally have unique identifying characteristics. As such, multi-class image classification with neural networks is a popular avenue for classifying skin diseases. Using the first dataset (17), baseline experiments were run with four different neural networks models including VGG, Resnet, Efficientnet, and Vision Transformer to classify 10 different skin diseases (classes).

Image augmentation techniques have facilitated the generation of synthetic dermatological images, augmenting existing datasets and enhancing the diversity and representativeness of training data. This approach helps mitigate the challenges of dataset imbalance and improves the robustness of AI models to variations in skin types, lighting conditions, and camera quality. Images were initially augmented through several standard techniques such as color jitter, gaussian blur, horizontal/vertical flip, and image resizing. Observing that a majority of “what is important” in the image is in the center of the image as well as the fact that skin diseases are orientation-agnostic, random cropping and random rotation image augmentations were also added. It is worth noting that, instead of directly random cropping to the desired resolution, a resize was first performed to a size slightly larger than the crop size to retain most of the image proportions. The Resnet-50 model was trained from scratch to determine the optimal data augmentation techniques on an 80: 20 train/ validation split. The first dataset (17) was selected in part because the class distribution was more or less even across the ten classes, simplifying the training process as there would be less worry of overfitting on one particular class. This test establishes a baseline performance that can be used for comparison with the new dataset proposed in this study.

Now that baselines for the first dataset (17) have been established, the aggregated new image dataset combines three datasets (17-19) for a total of 26 classes, 16 more than the baseline dataset (10 classes). Due to the new dataset’s class imbalance, reaching model convergence would be significantly harder than on the original dataset. To initially combat this issue, weighted random sampling was introduced which, instead of iterating through the whole train dataset per epoch, a weighted random sample was selected with the chance of an image from a particular class being selected as the inverse of the frequency. In this way, classes with less data are seen more often, reducing the effect of class imbalance. The same models from the baseline study are re-evaluated on this new dataset and the performance was compared.

To further combat class imbalance, transfer learning was used. Transfer learning occurs when 100% of the feature extractor is frozen and only the classifier is trained on a new dataset. In this case, the feature extractor is initialized to ImageNet weights. By using transfer learning, the model doesn’t need to learn a feature extractor and classifier simultaneously. This is especially useful in an imbalanced dataset as a trainable feature extractor would only learn the relevant features in the most common classes. Instead, a pretrained feature extractor would already understand what features are important in an image and enable the classifier to decide what disease these extracted features are related to.

However, directly using ImageNet-trained feature extractor weights has some issues. This is because the data distribution for the pretrained dataset, ImageNet, is significantly different from the current use case. While ImageNet contains pictures of dogs, cats, and other easily identifiable objects, the current dataset contains only skin disease images which have minute differences that correspond to vastly different diseases. As a result, what may be a distinguishing feature between a dog and a cat can’t be used to distinguish between melanoma and basal cell carcinoma. To address this, various percentages of the model layers were unfrozen (as compared to completely frozen in previous experiment) and allowed to be trained in order to determine the optimal percent at which the model would achieve the highest accuracy. By doing so, the model would be able to utilize the high level, class- agnostic, features learned through pretraining such as the outlines of shapes which are commonly found in the initial layers of the feature extractor. The model would then only need to adjust the later layers of the feature extractor which are responsible for low level features that are specific to the task at hand.

To further improve performance, image resolution was also adjusted. Since higher resolution images offer more visual cues of what the skin disease is, various models at 75% pretrained were evaluated. Beginning with the initial resolution of 224 x 224 pixels, the images were enlarged up to 528 x 528. At the initial 224 x 224 resolution used in previous works, the original image would need to be shrunk by approximately 80%, significantly reducing the number of features available. By increasing the resolution, the resulting image would have more features available for the model to discern using which might detract from performance as well since many of these features could be extraneous and not related to a disease. This study aims to determine the optimal resolution for images on this new dataset.

In conventional patient- doctor interactions, not only is the picture of the afflicted skin region available to the doctor but also a patient narrative that describes what symptoms the patient is experiencing. This information is important as some diseases may look the same from a physical inspection but cause the patient to exhibit different symptoms which are not apparent from an image. Therefore, based on the symptoms, the doctor should also be able to determine what type of skin disease the patient has. In the context of machine learning, this could be considered a multi class sequence classification task which large language models (LLM) excel at. Thus, three state of the art large language models (Llama-7B (20), Falcon-7B (21), Mistral-7B (22)) were fine- tuned on this dataset. This fine- tuning step is necessary in order for the model to understand what a proper sentence entails (from pretraining) as well as what specific keywords are associated with a particular disease. Due to computational constraints associated with such large models, LLMs larger than 7B were unable to be run and the three selected LLMs were finetuned using Low Rank Adaptation (LoRA). By using LoRA, only a small, low rank, subset of the model’s parameters (7 - 10%) need to be fine-tuned which enables the models to fit in consumer-grade GPUs. The dataset itself is split 70: 30 train/ validation due to its small size.

Along with the patient narrative text block input for sequence classification, additional supporting information was also added. As LLMs are trained on millions of corpuses that describe hundreds of diseases, it may be difficult for them to associate the presented symptoms to only the subset used in this study. As such, it makes sense to restrict the potential set of diseases the LLM must choose from to the subset by presenting the LLM, in its prompt, the available disease options. Initially, an explicit options mapping was concatenated at the end of each patient narrative text block which mapped each disease to its associated value such as Dermatofibroma to “1” and Lichen Planus to “7”. By explicitly adding this mapping, the LLM would only need to know the disease associated with the narrative as the mapping is given instead of learning both the disease and then mapping. Additionally, an implicit options list was also explored which concatenated only the possible diseases without their associated values as shown in Table 8.

Inspired by Chain of Thought prompting, a popular prompting technique that breaks down a complex reasoning task into intermediate steps for the LLM to take, Chain of Options breaks down the task during fine tuning instead of inference. This allows for more control in the reasoning process as the LLM’s performance could be evaluated during intermediate steps instead of only at the end. At a high level, Chain of Options incrementally removes k options from the options list based on which diseases are the most unlikely. Gradually, the options list would decrease until there are less than k diseases left at which point the top prediction is considered the final prediction. This “narrowing down” process reframes the task from needing to pick the most likely disease out of 26 options, which could be difficult if many diseases have similar symptoms, into a task to simply determine which k diseases are the least likely. This is a simpler reframe as picking the least likely diseases could be from just a single symptom mismatch. During training, random combinations of diseases of varying lengths are concatenated in the same way as the options list was. This randomness simulates the various instances of the chain that could potentially propagate through while also adding some noise to the input to reduce overfitting.

Four base models were evaluated on the aggregated new image dataset which contains 36995 images spanning 26 classes. Their results are compared to the first dataset (10 classes) in Table 5. Across the board, model performance decreased with VGG- 11 and ViT failing to converge and Resnet- 50 and Efficientnet- B3 accuracy dropping by at least 10%, proving that this dataset, due to its class imbalance, was significantly harder than the original dataset.

Transfer Learning

Since VGG-11 and ViT failed to converge when trained from scratch on this dataset, the transfer learning paradigm was used. Using this method, the training speed of the models significantly increased with most models reaching convergence in around 4 hours as opposed to 8 hours in the from scratch case. ViT was able to converge although VGG-11 and 16 still failed to do so. This makes sense as ViT isn’t able to leverage the inductive bias that convolutional neural networks (VGG, Resnet, and EfficientNet) have inherently baked in since ViT uses transformer layers as opposed to convolutional layers. Therefore, when the dataset is not only small but also imbalanced, ViT struggles to learn as it must learn inductive biases toward images as well as the classes of the images simultaneously. By using transfer learning, ViT has already learned the inductive biases toward images and would only need to learn the image classes for a new dataset. VGG, on the other hand, struggles from the “vanishing gradient” problem. Due to the depth of the network, gradient updates to initial layers of the network become very small, significantly increasing the training time. However, without such a large network, the intricacies of the dataset are more difficult to learn as the model isn’t able to fit as complex of functions.

Table 6 compares the epochs and accuracies between not pretrained and pre-trained with 100% feature extractor frozen (transfer learning) models. Directly using pre-trained feature extractors lead to suboptimal performance for models that are able to initially converge for both Resnet and Efficientnet models. However, pre-training helps improve performance for all other models (VGG, ViT).

Fine-Tuning of Pre-trained Resnet Model

For this task, the best performing model (Resnet-50) across both not pre-trained and 100% pre-trained was used and it was determined that 75% frozen parameters is optimal as shown in Figure 2. As the percent of frozen parameters increased, the number of epochs it took for the model to converge decreased. Similarly, as the percent of frozen parameters increased, so did the accuracy up to a certain point. Exceeding 75% frozen, the accuracy begins to decrease. Conceptually, this amount is the perfect balance between the high level, class- agnostic, features learned through pre-training and the low level, task- specific, features learned during fine-tuning.

Fine-Tuning of Image Resolution

The top performing model was found to be Resnet50 with images at 300 x 300 resolution. This model achieved 80.1% top-1 accuracy which is 8% higher than the baseline model. Further analysis shows that this model achieves 92.1% and 95.2% top-3 and top-5 accuracy, respectively. Increasing the resolution past 300 x 300, accuracy improvement saturates as the model accuracy is about the same between 300 x 300 and 528 x 528 resolution.

In conclusion, after optimizing the Resnet-50 model through image augmentation, fine-tuning, image resolution adjustment, the model with image resolution of 300x300 and 75% pre-trained on the new image dataset (36995 Images and 26 classes) achieved 80.1%, 92.1% and 95.2% for top-1, top-3 and top-5 accuracy, respectively as shown in Figure 3.

Choice/ Options Mapping The implicit options list outperformed the explicit map, proving that learning the mapping between disease and label is quite simple and that the mapping adds too much duplicate information which would actually hurt LLM performance. Furthermore, it demonstrates that LLMs perform better on multi- class sequence classification tasks when all the possible options are provided. By specifying which diseases, the LLM should consider through an explicit/ implicit options map, LLM performance is improved. In most cases, the implicit options map outperforms the explicit one with highest accuracy (89.7%) being achieved by Llama -7B and implicit options map.

Prediction Concatenation In order to utilize the predictions from the image model, they were concatenated onto the patient narrative in a similar way as choice/ options mapping. With this addition to the narrative, model accuracy increased by 20% from the baseline model for all three LLMs as shown in Table 9. Adding only the top-1 prediction had the highest improvement while top-3 and top-5 had slightly less. This is interesting because, as the number of provided predictions increased, the accuracy of the LLM decreased despite the likelihood of the correct class being in the predictions increasing as well. The same phenomenon occurs when we combine predictions and choice/ options mapping: although performance is better than just choice/ options mapping, performance decreases as the number of predictions increases.

Chain of Options To mitigate the issue of accuracy decreasing with more provided predictions, a novel training strategy was introduced called Chain of Options. By using Chain of Options, LLM performance for higher numbers of recommendations improved across all LLMs while maintaining at least a 90% accuracy as shown in Table 10.

Using resnet-50, image alone is only able to achieve  80% accuracy on the new dataset as discussed in the previous section. However, by combining both the top performing image model resnet-50 and the best LLM (Llama-7B), a final skin disease classification accuracy of 91% was achieved with the image model recommending the top 5 candidate diseases and the LLM adding these candidates to the input along with chain of options fine-tuning. This improved the overall accuracy as seen in Table 11.

In summary, the Resnet-50 model was applied to the new image dataset and the top-1 validation accuracy was 70% for 26 types of skin disease diagnosis. After further optimizing the model through various means such as fine-tuning, top-1 accuracy was improved to 80%. The best LLM, by itself, was able to achieve 76% accuracy. By combining the top performing image model with the top LLM, a final accuracy of 91% was achieved in Figure 4. This result outperforms human dermatologists in certain cases for over 26 types of skin disease diagnosis.