Capturing Local and Global Features in Medical Images by Using Ensemble CNN-Transformer

Medical imaging relies on a diverse array of techniques such as optical coherence tomography, computed tomography, and magnetic resonance imaging to generate detailed visual representations of the body’s internal structures [1]. The interpretation of these medical images by healthcare professionals is of paramount importance for guiding future medical interventions. However, the process of training and analyzing the proficiency of human experts is time-consuming [2]. In the absence of skilled radiologists, advanced computer vision (CV) models have emerged as invaluable tools capable of swiftly providing highly accurate diagnostic assessments. Computer vision, as a prominent subset of deep learning field, encompasses a wide range of tasks including classification, segmentation, and denoising. Its overarching objective is to extract meaningful features from diverse visual inputs, encompassing photographs, videos, and various visual data sources. These computer vision models offer the potential to revolutionize the field of medical imaging by expediting the diagnosis process, reducing human error, and improving patient outcomes. As technology continues to advance, the synergy between medical professionals and machine intelligence promises to enhance the efficiency and accuracy of healthcare diagnostics and treatment planning. The Transformer and CNN are two of the most commonly used classification models for images, both of which are used in image categorization.

CNNs, which have been a stalwart in the Computer Vision (CV) field for decades, excel at capturing hierarchical spatial information with additional convolutions [3, 4]. CNNs typically comprise convolution layers, dropout layers, fully connected layers, and other similar components. In contrast, the transformer, a relatively newcomer to the CV landscape compared to CNNs, originally emerged within the realm of natural language processing and revolves around the concept of attention, also referred to as self-attention [5]. As elucidated by Saadati in [6], attention involves the ability to connect various positions within a sequence to compute its representation. The pivotal distinction between CNN-based and transformer-based approaches lies in their focus: the former primarily consolidates local properties, whereas the latter takes a more global perspective [6].

Our CETC architecture is structured around three fundamental components: a transformer classification block (TCB), a transposed-convolutional decoder block (TDB), and a convolutional encoder block (CEB). This amalgamation is designed to deliver a holistic feature extraction and classification process. Here’s a detailed overview of how each element operates within CETC: Convolutional Encoder Block (CEB): The CEB plays a pivotal role in the system by capturing features across a spectrum of local scales. Through the utilization of convolutional layers, it excels in discerning intricate patterns and details in the input data. Transposed-Convolutional Decoder Block (TDB): Complementing the CEB, the TDB is responsible for decoding the captured features, returning them to their original scale. Subsequently, it integrates these decoded features using a set of three ensemble coefficients, thereby facilitating the effective amalgamation of multi-local information. Transformer Classification Block (TCB): In the final phase of CETC, the aggregated features, now available at both multi-local and global scales, undergo further processing through transformer-based mechanisms. This phase is instrumental in extracting global features, which are essential for comprehensive understanding and classification. The beauty of CETC lies in its seamless ability to capture features at both high and low frequency, thereby ensuring a comprehensive representation of the input data. Furthermore, our introduction of ensemble coefficients enables precise control over the contribution of local features across different scales, enhancing the model’s adaptability and performance. The robustness and effectiveness of CETC are evident in the evaluation results obtained from two prominent datasets: there are two datasets that are known to be available in COVID-19 radiography /cite[covid19radio, wang2020covid] and COVIDx CXR-3 [7]. It is consistently shown that CETC outperforms other models that are considered to be state-of-the-art (SOTA) across a wide range of evaluation metrics. In summary, our primary contributions and innovations can be encapsulated as follows:

• •

  The purpose of this study is to design a new CETC model that combines CNNs with transformers for the classification of medical images.

• •

  The developed CETC is capable of capturing both local and global features at the same time. At various scales, local features can be managed by adjusting the ensemble coefficients so that local contributions to the ensemble are optimized.

• •

  In our experiments on two open access datasets, we have found that our CETC excels SOTA methods in all evaluating metrics on both datasets.

The remaining sections of this paper are structured as follows: Section 2, titled "Related Work," provides an overview of previously published studies on CNN and transformer models, specifically focusing on their applications in medical image classification. Section 3, labeled "Materials and Methods," presents a summary of the datasets utilized in our research, as well as the implementation details, including architectural design, experimental settings, and evaluation metrics. In Section 4, labeled "Results and Analysis," we present the outcomes of our study on two public COVID-19 datasets, along with a comprehensive analysis of the findings. Furthermore, we conducted additional ablation experiments. Finally, in Section 5, entitled "Conclusions," we provide a summary of our work and discuss future research directions.

The CETC model’s detailed architecture is illustrated in Figure 1. It comprises three sub-encoders, denoted as $\mathrm{SE}_{1}$, $\mathrm{SE}_{2}$, and $\mathrm{SE}_{3}$. Each of these sub-encoders leverages components from well-established neural network architectures such as VGGNet [8], ResNet, and MobileNet as their core frameworks. To ensure uniformity with the $224\times 224$ mount high level features, the output scales of $\mathrm{SE}_{1}$, $\mathrm{SE}_{2}$, and $\mathrm{SE}_{3}$ are set to $112\times 112$, $28\times 28$, and $56\times 56$ respectively. This design choice allows for the capture of local features at different scales, specifically $112\times 112$, $28\times 28$, and $56\times 56$ scales. Moving on to the Transposed-Convolutional Decoder Block (TDB), it incorporates three sub-decoders denoted as $\mathrm{SD}_{1}$, $\mathrm{SD}_{2}$, and $\mathrm{SD}_{3}$, each specifically designed with an output dimension of 224 × 224 to match the input dimensions of $\mathrm{SE}_{1}$, $\mathrm{SE}_{2}$, and $\mathrm{SE}_{3}$. To consolidate the features produced by these sub-decoders, we employ an innovative feature map-level summation operation, utilizing the ensemble coefficients we have introduced. Lastly, the TCB employs a Swin Transformer (SwT) as its core architecture, complemented by a specially tailored prediction head. This block takes the summed feature map as input and is proficient in capturing the global features at the 224 × 224 scale. Through the synergistic combination of the CEB, TDB, and TCB, the CETC model excels in effectively capturing features spanning from 28 × 28 to 224 × 224 scales, encompassing both local and global aspects.

In the CEB, $\mathrm{SE}_{1}$ originates from the upper 17 sub-layers of VGGNet, whereas $\mathrm{SE}_{2}$ and $\mathrm{SE}_{3}$ are derived from the upper 5 and 2 sub-layers of MobileNet and ResNet, respectively.

The TDB comprises distinct elements for each sub-decoder. $\mathrm{SD}_{1}$ encompasses an upsampled convolutional layer, followed by a ReLU filter function. It also encompasses a $5\times 5$ transposed convolutional layer with an accompanying ReLU activation function, and a subsequent $5\times 5$ transposed convolutional layer. $\mathrm{SD}_{2}$ comprises of a $5\times 5$ transposed convolutional layer, succeeded by a ReLU activation function, and another $5\times 5$ transposed convolutional layer. On the other hand, $\mathrm{SD}_{3}$ consists of a solitary 4 × 4 transposed convolutional layer. The process of feature map-level summation is carried out with the involvement of ensemble coefficients denoted as $\alpha$, $\beta$, and $\gamma$, and the resultant output $y$ is computed as follows:

Here, $\alpha$, $\beta$, and $\gamma$ are coefficients ranging from 0 to 1, and their sum equals 1. ${FSD_{2}}$, ${FSD_{1}}$, and ${FSD_{3}}$ represent the output feature maps of $\mathrm{SD}_{2}$, $\mathrm{SD}_{1}$, and $\mathrm{SD}_{3}$, respectively.

In the TCB, a total of four levels are present, each featuring a different count of CSwT blocks. Each CSwT block is comprised of two components: one featuring a multi-head attention module employing standard windowing configurations (W-MSA), and the other employing shifted windowing configurations (SW-MSA). The MLP module consists of two layers, separated by a GELU activation function. The prediction head encompasses a linear layer with two output forks. The operation of the CSwT block can be summarized as follows:

Within this context, $\hat{\mathbf{z}}^{l}$ and $\mathbf{z}^{l}$ denote the output characteristics of the MLP and the W-MSA components within block $l$, respectively. It’s worth noting that attention computations involve the inclusion of relative position bias.

In this section, we initiate the discussion by examining the datasets utilized to train our enhanced Transformer model, and provide a summary of the experimental configurations. Following that, we conduct a comprehensive analysis of the experimental outcomes, here, by comparing the CETC model with the most recent advances in the task of the classification of medical images, we are able to assess the effectiveness of the model.

In this paper, we present the Controllable Ensemble Transformer and CNN network model for analysing medical images in a controlled way. Using the CETC model, you can obtain both high and low level features in medical images by combining the strengths of convolutional neural networks and transformers. It consists of three blocks, namely a CEB, a TDB, and a TCB. The CETC model conducted better in terms of various evaluation metrics than existing SOTA models based on experimental results conducted on publicly available COVID-19 datasets. This highlights the potential of the CETC model for accurate and efficient analysis of medical images. In our future endeavors, we aim to broaden the scope of the proposed method beyond the classification task, while simultaneously enhancing its capacity for feature extraction.