Deep-learning models in medical image analysis:
Detection of esophagitis from the Kvasir Dataset

CNN is a deep learning method specialized for image recognition. It is widely used for identifying lesion sites in medical images. It combines a convolutional layer with a pooling layer and finally iterates through all the combined layers to generate the results. In this study, we compared the results of different CNN models used for site identification in medical images. The CNN models used included GoogleNet and ResNet, the successive winning models of ILSVRC, and MobileNet V2 and MobileNet V3, which have attracted considerable attention in recent years because of their small computational and memory.

The CNN models were compared with XAI methods Grad-CAM and SHAP. The Discussion section explains the results obtained using these techniques.

The Kvasir dataset is a collection of endoscopic images of the gastrointestinal tract. It was annotated and validated by certified endoscopists. The dataset was made available in the fall of 2017 through the Medical Multimedia Challenge provided by MediaEval. It includes anatomical landmarks (pylorus, z-line, and cecum), disease states (esophagitis, ulcerative colitis, and polyps), and medical procedures (dyed lifted polyps and dyed resection margins). The resolution of the images from the Kvasir dataset with these eight classes varies from 720$\times$576 pixels to 1920$\times$1072 pixels. Each image has a different shooting angle, resolution, brightness, magnification, and center point.

Image prepossessing was performed before training the models. Edge artifacts and annotations that interfere with learning during the analysis of medical images were removed. A mask image was created, where pixels with luminance values below a certain threshold were set to 0. The opening process was applied to the mask image to remove the annotations. The image was cropped using this final mask image to obtain the target area. This process was performed on all data.

Each image in the dataset has a different resolution. All images were resized to 224$\times$224 pixels by bilinear completion and optimized for deep learning input. In addition to these processes, data augmentation was performed on the data used for learning. We applied two types of data augmentation: horizontal and vertical flip.

A total of 1000 image data sets containing z-lines and esophagitis were partitioned into test, training, and validation data. First, 25% (n = 250) of the total data were randomly selected to generate test data. Of the remaining data (75%, n = 750), 50% (n = 500) was used for training and 25% (n = 250) for validation.

The inner loop consisted of training and validation data. The model was trained using the training data, and parameters such as the optimal number of epochs were determined using the validation data. Thus, four training models were generated. The test data of each model were evaluated, and the average of discrimination accuracy of the four times was used as the evaluation value of the CNN model. The test, training, and validation data were each partitioned to maintain the class proportions.

PyTorch was used for the implementation of GoogLeNet, ResNet-50, MobileNet V2, and MobileNet V3. The initial values of all model parameters were pre-trained by ImageNet, and the models were trained by fine tuning.

For all models, the Adam optimizer was used for training. The batch size was five, and the maximum number of epochs was 100. The cross-entropy error shown in equation (1) was used as the loss function.

Five evaluation indices were used in this experiment: accuracy, precision, recall, specificity, and F1-score. These metrics were calculated using the confusion matrix shown in Table 1.

In this experiment, the z-line and esophagitis were judged as the negative and positive classes, respectively. In other words, data judged to be esophagitis and z-line by the learning model were designated true positive (TP) and false negative (FN), respectively. Meanwhile, data determined to be esophagitis and z-line by the training model were designated false positive (FP) and true negative (TN), respectively. Based on the values of TP, FP, TN, and FN obtained from the confusion matrix, the accuracy, precision, recall, specificity, and F1-score of the models were calculated using Equations(2) to (6).

The evaluation indices obtained from the experiments are shown in Table 2.

The F1-score results in Table 2 show that GoogLeNet was the best among the four models. In other words, GoogLeNet was more reliable in predicting esophagitis than the other models. Meanwhile, MobileNet V3 showed the highest precision and specificity. In other words, MobileNet V3 was the most accurate among the tested models for z-line prediction. From a medical point of view, an ideal model should be likely to distinguish esophagitis with severe symptoms from the z-line.

The average of TP rate were 0.950, 0.923, 0.892, and 0.841 for MobileNet V3, MobileNet V2, GoogLeNet, and ResNet-50, respectively. MobileNet V3 predicted esophagitis with more confidence than the other models.

Grad-CAM and SHAP were applied to the learned model, and what kind of the model was created was discussed.

Figure1 shows an example of the image results in the case of TP predicted by GoogLeNet. In the Grad-CAM results, red indicates the most potent activation, and blue indicates the weakest activation. In the SHAP results, the SHAP values of the patches were computed and rendered in a color map: a positive SHHAP value (red) indicates that the class is supported. By contrast, a negative SHAP value (blue) indicates that the class is rejected.

Tearing the esophageal mucosa against the z-line is a feature of esophagitis. According to Figure1, the results of Grad-CAM and SHAP showed that the learned model of GoogLeNet can makes predictions focusing on the clinically significant aspects of esophagitis images. The GoogLeNet model learned the findings that are important for diagnosing esophagitis. Comparison results showed that SHAP captured the location of multiple mucosal tears in the image more accurately than Grad-CAM.

Figure2 shows the results of applying Grad-CAM and SHAP in the FN case. The following can be observed from the results of Grad-CAM and SHAP for Figure2, respectively. In the Grad-CAM results, most areas in the image are shown as activated regions. Areas that provide the basis for the prediction are difficult to identify because of the gradient saturation in the Grad-CAM calculation. In the SHAP results, the inflammatory areas of the input image are indicated by blue pixels. Blue pixels indicate features that have a negative contribution to the prediction. In other words, although the model incorrectly identified esophagitis as a z-line, the model recognized that areas in the image negatively contributed to the z-line decision.

One hundred images were determined to be TP in the MobileNet V3 model. The SHAP results for the images judged to have the highest and lowest probabilities of being esophagitis are shown in Figure3.

As shown in Figure3, in cases with a high prediction probability, some features may have a negative contribution to the prediction. Many features showing negative contributions can be identified in the images with low prediction probability for Figure3. In this case, the prediction probability may be low.