Multiscale Encoder and Omni-Dimensional Dynamic Convolution Enrichment in nnU-Net for Brain Tumor Segmentation

The dataset comprises a comprehensive collection of 5,880 MRI scans from a cohort of 1,470 patients diagnosed with brain diffuse glioma [3]. Among these scans, 1,251 have been designated for training purposes, while an additional 219 scans are allocated for validation. This dataset is consistent with the data meticulously compiled for the BraTS 2021 Challenge. This dataset focuses on glioblastoma (GBM), which stands out as the most prevalent and aggressive form of brain-originating cancer. With a grade classification of IV, GBMs display an extraordinary degree of heterogeneity in terms of their appearance, morphology, and histology. Typically originating in the cerebral white matter, these tumors exhibit rapid growth and can attain considerable sizes before manifesting noticeable symptoms. Given the challenging nature of GBM and its dire prognosis, with a median survival rate of approximately 15 months, the comprehensive dataset is a vital resource for advancing research, diagnosis, and treatment strategies in the field.

With a total training cohort size of 60 cases, the Sub-Saharan Africa dataset [1] represents a specialized collection of glioma cases among patients from the Sub-Saharan Africa region. This dataset stands out due to its distinct characteristics, arising from the utilization of lower-grade MRI technology and the limited availability of MRI scanners in the region. Consequently, the MRI scans included exhibit diminished image contrast and resolution. This reduction in clarity poses a challenge by obscuring the distinct features present in the aforementioned dataset and introducing intricacies in the segmentation process. Notably, patients within this dataset from the Sub-Saharan Africa region often present with comorbidities like HIV/AIDS, malaria, and malnutrition. These underlying health conditions influence the appearance of brain tumors in MRI scans, further complicating the diagnostic process.

The dataset contains an extensive collection of cases, comprising 1,000 instances designated for training and an extra 141 cases set aside for validation. These cases are centered around meningiomas [11], a specific type of tumor that originates from the meninges. These meninges are protective layers that cover the brain and spinal cord. Meningiomas are noteworthy since they represent the most frequently occurring primary tumors within the brain of adults. Their significant risks emphasize the clinical importance of these tumors in terms of health complications and mortality. This dataset was introduced as part of the BraTS challenge in 2023.

The Brain Metastases dataset’s [15] training segment comprises 165 samples. In comparison, an additional 31 samples are included in the validation set, providing a comprehensive collection of multiparametric MRI (mpMRI) scans focusing on brain tumors. Diverging from the characteristics of gliomas, which tend to be more readily detectable in their initial scans due to their larger size, brain metastases exhibit a distinctive heterogeneity in their dimensions. These metastatic tumors manifest across a spectrum of sizes, often presenting as smaller lesions within the brain. Notably, these metastatic growths possess the capacity to emerge at various locations throughout the brain, contributing to the diversity of their appearances.

A noteworthy aspect of this dataset is the prevalence of brain metastases measuring less than 10 mm in diameter. These smaller lesions stand out in terms of their frequency, underscoring their clinical significance. Unlike their larger counterparts, these diminutive metastases have been observed to surpass others in terms of occurrence.

Despite certain similarities between pediatric tumors [10] and adult tumors, notable differences exist in their appearance in both imaging and clinical contexts. Both high-grade gliomas, such as GBMs, and pediatric diffuse midline gliomas (DMGs) have limited survival rates. Interestingly, DMGs are roughly three times rarer than GBMs. While GBMs typically emerge in the frontal or temporal lobes in the age of 60s, DMGs are predominantly found within pons and are frequently identified in the age range of 5 and 10.

GBMs are typically characterized by the presence of an enhancing tumor region in post-gadolinium T1-weighted MRI scans, accompanied by necrotic regions. In contrast, these imaging characteristics are less pronounced or distinct in DMGs. Pediatric brain tumors are more inclined to be low-grade gliomas, generally displaying slower growth rates compared to those in adults. The dataset at hand comprises 99 training samples and 45 validation samples.

Before inputting the data into our model, we apply multiple data augmentation techniques to enhance the robustness of the model. These random augmentations within a specific range include crop, zoom, flip, noise, blur, brightness, and contrast. This ensures that the model can generalize better to various spatial transformations and variations in the input data.

We extract patches of size $5$ x $128$ x $128$ x $128$ from the original brain scan images. These patches capture localized features and enable the model to focus on specific regions of interest. This patch-based approach also aids in reducing memory requirements and computational complexity during training and inference.

Our proposed model architecture consists of two encoder layers, each inspired by the nnU-Net [7] framework. Instead of using conventional Conv3D layers, we employ Omni-dimensional Dynamic Convolution 2D (ODConv2D) layers, which we extend for use with 3D images and refer to as ODConv3D. ODConv3D integrates a multi-dimensional attention mechanism utilizing a parallel strategy. This empowers us to acquire complementary attention patterns for convolutional kernels across all four dimensions of the kernel space. This enriched dynamic convolution design enables the model to effectively capture intricate spatial, channel-wise, and temporal information.

Our multi-modal attention strategy enhances the model’s ability to leverage complementary information from different scales. We provide the original input image and a downsampled version of the same image as inputs to the model. This enables the model to capture fine-grained details and broader contextual information simultaneously. Separate encoders process each of these inputs, and the attention mechanism is then applied to fuse the encoded features. This approach encourages the model to focus on informative regions across scales, contributing to enhanced feature representation.

The decoder of our model is similar to the nnU-Net’s decoder architecture. It reconstructs the fused encoded features to generate a comprehensive feature map that preserves spatial information. This map is then processed to yield the final predicted segmentation.

Post-processing is applied to the output feature map to refine the predicted segmentation. This includes steps such as thresholding, morphological operations, and connected component analysis to remove noise and ensure coherent tumor regions.

To evaluate the accuracy and efficacy of our proposed model, we employ two performance metrics i.e., Dice coefficient and Hausdorff distance. These metrics provide quantitative measures of the segmentation quality and the model’s ability to delineate tumor boundaries accurately.

We evaluate the baseline performance of the nnU-Net architecture on the BraTS-Africa Dataset. As shown in Table 1, the nnU-Net achieves a Dice coefficient of 0.8980 for overall segmentation. Lesion ET, TC, and WT achieve Dice coefficients of 0.8354, 0.8485, and 0.6578, respectively.

We investigate the effects of integrating the Multiscale and ODConv3D into the nnU-Net architecture on the BraTS-Africa Dataset. The nnU-Net + Multiscale + ODConv3D model achieves an enhanced Dice coefficient of 0.9092 for overall segmentation. However, individual lesion subregions show varying outcomes, with Lesion ET experiencing a slightly decreased Dice coefficient.

When we replace the labels of the nnU-Net model by the nnU-Net + Multiscale + ODConv3D model’s Label 2 for the BraTS-Africa Dataset, this leads to an increase in Lesion WT from 0.65 to 0.75 as shown in table 1, improving the overall accuracy. Our model’s performance is highlighted in Table 2, demonstrating competitive results. Across all lesion subtypes, our model achieves Lesion ET scores of 0.8354, Lesion TC scores of 0.8485, and Lesion WT scores of 0.7564.

We begin by evaluating the baseline performance of the nnU-Net architecture on the Brain Metastases Dataset. As depicted in Table 3, the nnU-Net achieves a Dice coefficient of 0.7675 for overall segmentation. The performance varies across lesion subregions, with Lesion ET, Lesion TC, and Lesion WT achieving Dice coefficients of 0.5157, 0.5105, and 0.4656, respectively.

We assess the effectiveness of Omni-dimensional Dynamic Convolution (ODConv3D) by introducing the nnU-Net + ODConv3D model. This model replaces traditional Conv layers with ODConv3D layers for enhanced feature extraction. The results in Table 3 demonstrate that ODConv3D leads to an improved Dice coefficient of 0.7953 for overall segmentation. This improvement extends to lesion subregions, with Lesion ET, Lesion TC, and Lesion WT achieving Dice coefficients of 0.5364, 0.5451, and 0.5141, respectively.

We explore the benefit of a multiscale strategy by introducing the nnU-Net + Multiscale model. This approach leverages multiple scales of the input data to improve feature representation. As indicated in Table 3, the multiscale strategy yields a Dice coefficient of 0.7771 for overall segmentation. Lesion TC substantially improves with a Dice coefficient of 0.5737, compared to other subregions.

The culmination of our model’s innovation comes with the nnU-Net + Multiscale + ODConv3D configuration. Combining the multiscale strategy and ODConv3D layers yields exceptional results, evident in the overall Dice coefficient of 0.8188. This combined approach significantly enhances the segmentation performance across lesion subregions, with Dice coefficients of 0.5896, 0.6406, and 0.5555 for Lesion ET, Lesion TC, and Lesion WT, respectively. Table 4 showcases our model’s performance, underscoring its competitive results. Our model consistently achieves good scores across all lesion subtypes, including Lesion ET with a score of 0.5896, Lesion TC with a score of 0.6406, and Lesion WT with a score of 0.5648.