Disease-oriented image embedding with pseudo-scanner standardization for content-based image retrieval on 3D brain MRI

We begin by presenting the issues to be solved in this paper. As mentioned above, to realize CBIR for brain MRI, Onga et al. proposed a new technique called disease-oriented data concentration with metric learning (DDCML), which acquires low-dimensional representations of 3D brain MR images that are focused on disease features rather than the features of the subject’s brain shape [9]. DDCML is composed of 3D convolutional autoencoders (3D-CAE) effectively combined with deep metric learning. Thanks to its metric learning, DDCML could acquire reasonable low-dimensional representations for unlearned diseases according to their severity, demonstrating the feasibility of CBIR for brain MR images. However, we found that such representations are highly sensitive to differences in datasets (i.e., differences in imaging environments, scanners, protocols, etc.), which is a serious challenge for CBIR.

Figure 1 shows the low-dimensional distribution obtained by DDCML and visualized by t-SNE [29]. Here, DDCML was trained on Alzheimer’s disease (AD) and healthy cases (clinically normal; CN) in the ADNI2 dataset and evaluated ADNI2 cases not used for training and healthy control (Control – equivalent to CN) and Parkinson’s disease (PD) cases in the untrained PPMI dataset. From the perspective of CBIR, it is desirable to obtain similar low-dimensional representations for CN and Control. However, it can be confirmed that the obtained low-dimensional representations are more affected by the differences in the environment (dataset) than by the disease. As mentioned above, differences in imaging environments, including scanners, are a major problem in multi-center and time series analysis, and inconsistent low-dimensional representations because of such differences in datasets are a fatal problem in CBIR implementation. The purpose of this paper is reducing these differences and to obtain a low-dimensional representation that better captures the characteristics of the disease and is suitable for appropriate CBIR.

In studies dealing with multi-site and long-term data, it is undoubtedly important to reduce non-biological bias originating from differences among sites and datasets. Since the methods of Moyer et. al.[25] and Dinsdale et al. [26] are theoretical and straightforward learning method that utilizes images of the target site to achieve data harmonization, their robustness to unexpected input (i.e. from another site or dataset) is questionable. Therefore, in principle, the images of all target sites (scanners, protocols) need to be learned in advance. Since CBIR requires more consideration of the use of images taken in the past, the number of environments that need to be addressed can be larger than for general data harmonization. It will be more difficult to implement a harmonization method that learns all the data of multiple environments in advance. Therefore, in contrast to their approaches, we aim to achieve data harmonization by converting images taken in each environment into images that can be regarded as having been taken in one predetermined ”standard” environment (e.g., the scanner currently used primarily at each site). However, in addition to the problems described above, it is practically impossible to build an image converter for each environment.

With this background, we have developed a framework that combines CycleGAN, which realizes robust image transformation, with deep metric learning to achieve a certain degree of harmonization even for images in untrained environments. In this paper, we validate the feasibility of our framework, which converts MR images captured in various environments into pseudo standard environment images using only one type of image converter.

The pre-processing component performs the necessary pre-processing for future image scanner standardization processing and low-dimensional embedding acquisition processing. Specifically, for all 3D brain MR image data, skull stripping was performed using a multi-atlas label-fusion algorithm implemented in the MRICloud [30]. The skull-stripped images were linearly aligned to the JHU-MNI space using a 12-parameters affine transformation function implemented in the MRICloud, resulting in aligned brain images. This feature makes a significant contribution to the realization of the proposed PSS in the next stage. It is important to note here that since brain volume information is the feature that contributes most to the prediction of the dataset [16], the alignment to a standard brain with this skull stripping technique should also contribute to the harmonization of the data. In addition, because the intensity and contrast of brain MR images are arbitrarily determined, there is a large inter-image variation. In brain MR image processing using machine learning, the variation in the average intensity confounds the results. Therefore, we standardized the intensity so that the average intensity value of each case was within mean $\mu=18$ and margin $\epsilon=1.0$ by performing an iterative gamma correction process, as in previous studies [31][9].

In this experiment, we used the ADNI2 and PPMI datasets, in which the vendor information of the scanners (Siemens [SI], GE Medical Systems [GE], Philips Medical Systems [PH]) was recorded along with the disease information. Statistics of those datasets used in the experiment are shown in Table I. We used Alzheimer’s disease (ADNI-AD or AD) and clinically normal cases (ADNI-CN) from ADNI2 dataset with vendor information. From the PPMI dataset, we used two types of labeled images, Parkinson’s disease (PD) and Control. We did not utilize the scanner information for this dataset in evaluating the versatility of the proposed method. Note that ADNI-CN and Control can be considered medically equivalent. Furthermore, PD is known to show little or no difference in MRI from healthy cases [32][33].

The ADNI and PPMI are longitudinal studies that include multiple time points, and the datasets contain multiple scans for each participant. To avoid duplication, one MRI was randomly selected from each participant. The MRICloud (https://mricloud.org/) was used to skull strip the T1-weighted MRIs and affine transform to the JHU-MNI atlas [34]. A neurologist with more than 20 years of experience in brain MRI research performed the quality control of the MRIs and removed MRIs that the MRICloud did not appropriately pre-process. Due to the neural network model used in the experiments, the skull-stripped and affine-transformed brain MR images were converted to 160$\times$160$\times$192 pixels after cropping the background area. Training and evaluation of the PSS network and embedding network were performed using five-fold cross validation. In the evaluation experiments described below,, the evaluation data of each fold is not included in the training data for either the PSS network or the embedding network. Note that even skilled and experienced neuroradiologists cannot separate PD from CN or Control by visual inspection of the T1-weighted images. Therefore, we did not expect these two conditions to be separable by unsupervised clustering methods even after applying the DI-PSS.

Figures 3a and 3b show the architecture of the generator ($G_{X},G_{Y}$) and the discriminator ($D_{X},D_{Y}$), respectively of the PSS network. They are basically the same as the original CycleGAN for 2D images. Since PSS is to reduce the bias caused by variations in scanners and scan parameters, the disease-related anatomical variations should be minimized in the training images. Therefore, we used only ADNI-CN cases, in which disease features do not appear in the brain structure, to train the PSS network. In this experiment, we chose the Siemens scanner as the standard scanner because it has largest market share, and we chose the GE scanner as the specific vendor of image conversion source. In other words, our PSS network is designed to convert CN images taken by GE scanners from the ADNI2-dataset (CN_GE) to synthetic images similar to those scanned by the Siemens scanners (CN_SI). We evaluated the applicability of the PSS to the diseased brain MRIs (AD and PD), as well as the generalizability to the non-GE scanners (see Section IV.D). In PSS network, we used coronal images for the training. The number of training images of each fold in the PSS network is (93+92)$\times$4/5 (5-fold CV)$\times$192 (slices).

Figure 4 shows the architecture of our 3D-CAE-based embedding network. Our embedding network embeds each 3D brain MR image into 150-dimensional vectors. The size of the MRIs handled by the embedding network is halved at each side, as in DDCML [9], to improve the learning efficiency. Note that the compression ratio of our embedding network is (80$\times$80$\times$96):150 = 4,096:1. The embedding network was trained and evaluated using ADNI2 and PPMI datasets with the five-fold cross-validation strategy.

As mentioned above, PD and CN cannot even be diagnosed from images by skilled neuroradiologists, so for training 3D-CAE to obtain low-dimensional representations, two classes of metric learning are used so that the representations of AD and (CN + Control) are separated. The low-dimensional representations of brain MR images are acquired by five-fold cross validation of 3D-CAE. In addition to AD, CN, and Control in each test fold, the low-dimensional representation of PD, which was not included in the training, is analyzed to quantitatively verify the effectiveness of the proposed DI-PSS evaluation.

To evaluate the effectiveness of the proposed DI-PSS framework, we evaluate the three following elements:

1. 1.

   Changes in MR images

2. 2.

   Distribution of the embedding.

3. 3.

   Clustering performance of the embedding.

In (1), we assess how the images are changed by our scanner standardization. We quantitatively evaluate the difference between the original (raw) image and the synthetic image with peak signal-to-noise ratio (PSNR), root mean squared error (RMSE), and structured similarity (SSIM). To ensure that the evaluation is not affected by differences in brain size, these evaluations were performed on brain regions only. Although MRICloud, which is used in skull stripping in this experiment, standardizes the brain size to the standard brain size, reducing the differences in brain size between cases, this method was adopted for a more rigorous evaluation.

In (2), we quantitatively examine the effect of PSS by analyzing the distribution of the obtained low-dimensional representations. Specifically, for each category (e.g., CN_SI, AD_GE) we investigate the following: (i) variation (i.e., standard deviation) of the embedding and (ii) the mean and standard deviation of the distance from each embedding to the centroid of a different category, where the distance between the centroids of ADNI-CN_Siemens (CN_SI) and ADNI-AD_Siemens (AD_SI) are normalized to 1. In addition, we visualize those distributions in 2D space using t-SNE [29] as supplemental results for intuitive understanding.

In (3), we evaluate the separability of the resulting embeddings. In this study, we performed spectral clustering [35] to assess its potential quality for CBIR. In the spectral clustering, we used a normalized graph Laplacian based on 10-nearest neighbor graphs with a general Gaussian-type similarity measure. We set the number of clusters to be two (AD vs. CN + Control + PD), which is the number of disease categories to be classified. Here, the consistency of the distance between the embedded data because of the difference in folds is solved by standardizing the distance between CN_SI and AD_SI per fold to be 1, as mentioned above.

The clustering performance was evaluated using two methodologies. The first was evaluation with six commonly used criteria (i.e., silhouette score, homogeneity, completeness, V-measure, adjusted Rand-index [ARI], and adjusted mutual information [AMI]) implemented on the scikit-learn machine learning library (https://scikit-learn.org/). The other is a diagnostic capability based on clustering results. Here, as with other clustering evaluations in the literature, we swap the columns so that each fold results in the optimal clustering result and then sum them.

Figure 5 shows an example of each MR image converted to an image taken on a pseudo-standard (= Siemens) scanner with PSS and the difference visualized. Table II summarizes the statistics of the degree of change in the images in the brain regions. Here, the background region was excluded from the calculation to eliminate the effect of differences in brain size. For the ADNI dataset, the differences obtained by the PSS image transformation were not significant between CN, AD, and scanner vendors, although the Philips scanners showed less variation on average. For the PPMI dataset that was not used for training, the change in the image because of PSS is clearly larger compared with ADNI (approx. $\times$ 1.5 in RMSE). In all categories, the amount of change because of PSS varied from case to case, but the PSS treatment did not cause any visually unnatural changes in the images.

Figure 6 shows the cumulative intensity changes of images by PSS in each category. This time, the background areas other than the brain are also included in the evaluation. The number of pixels where the intensity has not changed because of PSS exceeds 80% for all categories, indicating that no undesired intensity changes have occurred for the background (as also seen in Figures. 5 and 6). There is no significant difference in the distribution of intensity change by vendor, and the PPMI dataset has a larger amount of intensity change overall.

In this section, we compare the separation ability of the obtained low-dimensional embedding of MR images with and without PSS (baseline).

Tables V summarizes the clustering performance evaluated with six commonly used criteria. These are the silhouette score (silh), homogeneity score (homo), completeness score (comp), V-measure (harmonic mean of homogeneity and completeness; V) , ARI, and AMI implemented on the scikit-learn library. In each category, 1 is the best score and 0 is a score based on random clustering. It can be confirmed that PSS improved the clustering ability in all evaluation items.

Table VI is a summary of the clustering performance evaluated with the diagnostic ability. Table VI (a) is a confusion matrix. Here, the numbers of CN, Control and AD cases are the sum of each fold in the cross-validation. In each fold, we tested all PD cases (not included in the training), and the number was divided by five and rounded to the nearest whole number. Tables VIb and VIc summarize the diagnostic performance calculated from Table VI (a) without and with PD cases, respectively. It can be confirmed that PSS enhances the separation of AD and other categories (i.e., CN, Control and PD) in the low-dimensional representation.

PSS improved the diagnostic performance by about 6.2% (from 73.7 to 79.9%) for micro-accuracy and about 10.7% (from 63.8 to 74.5%) for macro-F1. The specificity for PD was also improved by 6.1% (from 69.7% to 75.8%).

Our PSS network transforms healthy cases taken with GE scanners to those taken with Siemens scanners. As can be seen from Figure 6 and Table II, the amount of change in the images because of PSS was almost the same for both AD and CN images in the ADNI dataset, including the Philips case.

The amount of conversion of the image for the PPMI dataset was larger than that for the ADNI dataset. This is thought to be due to the process of absorbing the differences in the datasets that exist in the image but are invisible to the eye. However, in all cases, the converted images have a natural appearance without destroying the brain structure. This can be objectively confirmed in SSIM, which evaluates the structural similarity on the image, maintains a high value. As discussed in detail below, PSS can reduce disease-specific variation in the resulting low-dimensional embedding, absorb differences among datasets and scanner vendors, and improve the separability of diseases. Given these factors, we can conclude that this PSS transformation was done properly.

This section discusses the effects of our DI-PSS framework from the perspective of CBIR implementation.

The recently proposed data harmonization methods for brain MR images by Moyer et al [25] and Dinsdale et al. [26] have been reported to be not only logically justified but also very effective. However, as mentioned above, these methods are difficult to apply to CBIR applications because images from all scanners are theoretically needed to train the model. Our DI-PSS is a new proposal to address these problems.

Although DI-PSS only learned the transformation from CN_GE to CN_Siemens, the improvement of the properties of the obtained embeddings was confirmed even for combinations that included other companies’ scanners, such as the Philips scanner, and different disease categories (AD) that were not included in the training. The results are evidence of proper data harmonization. We think this is due to the combination of MRICloud, an advanced skull stripping algorithm that performs geometric and volumetric positioning, and CycleGAN’s generic style transformation capabilities and distance metric learning, which make up the PSS network. Experiments with large-scale data from more diverse disease classes are needed, but in this experiment, we could confirm the possibility of obtaining effective scanner standardization by building one model that translates into a standard scanner.