CT Image Harmonization for Enhancing Radiomics Studies

The architecture of RadiomicGAN shown in Figure 1 consists of a generator $G$ and a discriminator $D$. The discriminator of RadiomicGAN $D$ is a fully convolutional neural network [14]. The encoder of the generator $G$ is constructed as a series of consecutive neural blocks containing both pre-trained and trainable layers inspired by the applications of pre-trained VGG for style transfer [9, 10]. Specifically, every neural block includes multiple pre-trained VGG layers (Figure 1, light blue), which remain frozen during network training, and a trainable layer (Figure 1, dark blue) that works as filter to extract and forward fine-to-coarse texture features to the corresponding decoding layer with a skip connection aiming to preserve the lost features during down-sampling [20]. For example, the first neural block in Figure 1 includes two 512-by-512 pre-trained layers and a 512-by-512 trainable layer. Both the first and the second layers are constricted by applying the convolution operation with stride=1 on the previous layer. The third layer in the same neural block, however, is trainable and is designed to filter out the redundant features and to propagate domain-specific information onto the corresponding decoding layer. Since we expect to standardize the texture features while keeping the shape features unchanged [9], only the first four groups of convolutional layers of a pre-trained VGG-19 network are adopted. RadiomicGAN is trained using the adversarial loss with L1 regularization [14].

Unlike natural images with pixel values ranging from 0 to 255 (i.e. 8-bit encoding), CT images follow the 16-bit encoding and thus have a much wider dynamic range. To effectively transfer the learned information from the 8-bit pixel domain to the 12 bit pixel domain, we introduce a new training strategy, called Dynamic Window-based Training (DWT). DWT is designed to exposes a deep learning model to the most suitable pixel range in each training iteration, where the lower bound and the upper bound of a window can be automatically updated according to the training results in the previous iterations. Compared to the traditional min-max approach that normalizes the whole HU range to $[0,1]$, DWT allows the model to be gradually exposed to the data points with more local intensity details. It enables the utilization of a VGG-19 trained with natural images in ImageNet [6] for CT image standardization. We introduce two DWT steps, namely fixed growing and dynamic selection, of DWT in the following text.

The dynamic selection approach is illustrated in Figure 2. Let $\mathcal{P}$ be a randomly selected subset of the training data $DS$ and $\mathcal{P^{\prime}}$ be its corresponding synthesized images. After each training epoch, we feed $P^{\prime}$ to the current discriminator $D$ and calculate a set of CAMs $C$ using Grad-CAM  [22]. During the calculation, $D$ remains freeze. For each image $p_{i}\in P^{\prime}$, the layer before the soft-max layer in $D$ indicates the probability of being a standard and non-standard image, which is used to calculate the CAM $c_{i}$ of $p_{i}$. CAM can be visualized as a heat-map where values close to $1$ are critical for determining the image being non-standard. With a user given threshold $th_{fail}$, we can determine the subareas of $p_{i}$ contributed most for predicting $p_{i}$ to be a non-standard image ($th_{fail}\in[0,1]$). For image $p_{i}$, if a pixel’s value in the CAM is greater than $th_{fail}$, the corresponding CT image pixel value is added to a pixel intensity list named $W_{i}$. All the pixels with their frequency equal to $1$ are discarded from $W_{i}$ considering them as image noise. This process is repeated by randomly selecting subsets of images from the training data. The window range is defined by merging all the pixel intensity lists. The window range is recalculated in every epoch and the model is trained based on the current window range.

In total 14,372 CT image slices from lung cancer patient scans and phantom scans were obtained using three different reconstruction kernels (Bl57, Bl64, and Br40) and four different slice thicknesses (0.5, 1, 1.5, 3mm) using the Siemens CT Somatom Force scanner. We adopted Bl64 kernel as the standard CT imaging protocol, since it has been widely used in clinical practice for lung cancer diagnosis [18]. Two testing datasets were prepared for RadiomicGAN performance evaluation. The first testing data were captured using the reconstruction kernel Bl57. The second testing data were captured using the reconstruction kernel Br40. Each test dataset contains 387 image slices and have paired target standard images (Bl64). HU number range was set to between -1024 and 1000 in the standardization process, since most pixel values belong to this range. Random weights were used during the network initialization phase. Maximum training epochs were set to 30 with the learning rate being 0.0001 with momentum 0.5.

Three state-of-the-art CT image standardization models, i.e. Choe et al. [4], GANai [18], and STAN-CT [21], were selected for performance comparison. All the models, including RadiomicGAN, were developed based on TensorFlow [1] and trained using the same training data.

Model performance was evaluated systematically at the whole image (DICOM) level and with randomly selected regions of interest (ROIs) in three HU ranges, including $[-800,-300]$, $[-100,250]$, and $[300,800]$. For each CT image or ROI, in total 1,401 radiomic features were extracted using IBEX [24].

We examined the radiomic features reproducibility performance using Concordance Correlation Coefficient(CCC) [16]. CCC represents the correlation between the standard and the non-standard image features in a given features class. CCC ranges from -1 to 1 and is the higher the better. We conclude that a radiomic feature is reproducible if the synthesized image is more than 85% similar to the corresponding standard image (i.e., $CCC>0.85$) [4, 25]. Furthermore, we evaluated image quality using Structural SIMilarity (SSIM) [23] and Peak Signal-to-Noise Ratio (PSNR) [15]. PSNR is defined as a logged ratio of the peak signal and the mean-square-error between the synthesized and the standard images [12] and is the higher the better. SSIM is defined as a co-relation between the synthesized and the standard images with values ranges between -1 to 1, and the value 1 indicates prefect structural similarity.

We compared RadiomicGAN with Choe et al, GANai, and STAN-CT using all the three evaluation metrics on both real patient data and phantom data. We considered three versions of RadiomicGAN. RadiomicGAN was trained using the proposed window training strategy where both the fixed growing approach and the dynamic selection approach were used consecutively. RadiomicGAN¹ was a variation trained using the fixed growing approach only, and RadiomicGAN² was another variation trained using the dynamic selection approach only.

Table 1 indicates the effectiveness of CT image standardization on ROIs with different HU ranges. RadiomicGAN¹, RadiomicGAN², and RadiomicGAN outperformed the models-to-compare on all ROIs. Here, the column named “Input” shows the performance of the images before standardization.

In Table 2, the PSNR scores show that for CT images captured with Br40 or Bl57 kernel, RadiomicGAN achieved the best performance.

The SSIM scores show that all the models performed well on preserving the structural information. The SSIM scores of RadiomicGAN were slightly higher than the others.

Figure 3 illustrates the performance of all the models compared using a case study. The non-standard and the standard were phantom CT images acquired with the Br40 and Bl64 kernels respectively. The tumor in the image was highlighted in a green box and was magnified in the right upper corner. The results of all the models were visualized as well. A visual inspection indicates that the tumor image synthesized with RadiomicGAN was the most similar to the standard image. The total number of reproducible features computed using CCC was 612 for RadiomicGAN, higher than all the compared methods (Choe et al: 183, GANai: 209, and STAN-CT: 307). The RadiomicGAN¹ and RadiomicGAN² had 487 and 512 reproducible features respectively, which were also better than the compared models. The PSNR score of RadiomicGAN was 33.23, clearly higher than the compared methods (Choe et al: 26.08, GANai: 26.14, STAN-CT: 26.07).

While RadiomicGAN has achieved satisfactory performance, it is critical to check whether the image standardization does help cancer diagnosis. Here, we aim to differentiate benign from malignant lung nodules using image features extracted from the standardized CT images. We extracted 630 image features from six CT scans of pediatric sarcoma patients who had undergone resection of a small pulmonary nodule. Three of the resected nodules were histologically confirmed to be malignant (metastatic disease), while the other three were benign. Radiomic evaluation of these nodules identified eight image features consistently high for malignant compared to benign nodules (Figure 4). In contrast, before image standardization, none of the image features can differentiate malignant and benign nodules. The results suggest that image standardization is critical for CT images acquired with different parameters.

Figure 5 illustrate the mechanism and its effect in model training. Figure 5 (a) shows the window size variation during training. In the first eight epochs, the model was trained using the fixed growing approach. In epoch 7, the training window range achieved its full dynamic range and shifted the training strategy from the fixed growing approach to the dynamic selection approach. During the dynamic selection approach the window shifted based on the hot-spots of the selected training data. In this phase, the window size reduced dynamically and the training process end at epoch 13 when a reasonable window was reached. Figure 5 (b) shows a sample image hot-spot during the dynamic selection training phase. The active parts (red areas), which is corresponding to the pixels most difficult to synthesize, reduced quickly, indicating that window training is an effective approach for adopting models trained with ImageNet into the medical imaging domain.