Analyzing Deep Learning Based Brain Tumor Segmentation with Missing MRI Modalities

The original mmGAN implementation is based on PyTorch. We have our mmGAN implementation based on TensorFlow to give researchers another choice and also to better integrate with DeepMedic for the MRI brain tumor segmentation with missing MRI sequences.

The current implementation uses a complex configuration file for various functions. It is difficult for users to understand its contents, or modify it according to local environments. The original authors’ version of the code mainly focuses on experiments, a lot of the components are used for preprocessing different datasets or doing measurements, which might cause confusion when read by other users of the code. The users may also have difficulties in using different datasets rather than BRATS18, or utilizing further the outcome of the network. We allow users to modify the preprocessing and training configurations easily, and make the model adjustable to different sequential datasets. All the processes are controllable through the command line, bringing convenience for batch processing. Users may first call preprocessing function to preprocess the data, and then train the model with either TensorFlow or PyTorch frameworks, after that they may use the test function to predict certain groups of images with all the missing sequence scenarios. The synthesised results can be used for downstream tasks such as tumor segmentation.

Using the original implementation, it can be difficult to manage data and achieve cross-validation, datasets have to be managed manually instead of automatically. With our implementation, users can easily separate the dataset into several folds and select which folds are used for training or validation.

In our code, we allow the slices of images of different patients to be shuffled randomly, thus the slices in each batch come from different patients, without certain order. The purpose is not only to avoid the model bias on certain sequences of continued slices but also to relieve the negative effect of continued damaged sequences that appears in the dataset. Additionally, the user may also choose to use different combinations of missing sequences within one batch, by changing the “full_random” parameter. For example, when setting the parameter into “False”, a batch with the size of 8 will have all the 8 groups of MRI images with the same channels to be set to zero; yet with the “True” setting, each group would have different channels to be the missing ones.

The image size of BRATS datasets is 240*240, the same as some tumor segmentation models’ requirements (such as DeepMedic), yet the input size of mmGAN is 256*256. One method is to pad (as shown in Figure 2) the BRATS images directly with zeros to the expected resolution, use them for synthesis, and then crop the 240*240 region out for DeepMedic; another is to crop the images with a bounding box and resize them into 256*256 to the dataset (as shown in Figure 3), using them for training, as [5] did, then resize it back, to the original resolution to suit the requirement of DeepMedic. The user may change the “operation” parameter between “padding” and “crop”.

During preprocessing, the original implementation requires around 60 GB of memory, which is unnecessary, and beyond the memory capacity of most PCs; the generated HDF5 files are not compressed, which also occupy large space in the disk; the data loader used during training may not have good support of multi-processing in some environments, but running within only one process it takes much more time for training. We remake the preprocess functions, compare to the original version, our version is much more memory efficient. The preprocess program runs with multi processes. For each patient, a mean of all the voxels (within a bounding box of the main region of the brain) is computed, then the value of each voxel is divided by this mean value for standardization. Images of only one patient are opened in the memory once, so there is no special requirement of memory size; because of the parallelism, time efficiency is ensured. We use the TensorFlow TF-Record dataset, which requires less memory and has better support for multi-processing. The method allows data to be compressed to take up less space in the disk but is still highly efficient to be loaded. The provided data loader of TensorFlow will automatically choose the number of processes to use and adapt the speed to increase efficiency. When running preprocessing functions, the images from patients will be arranged randomly into several TF-Record files, assigned in the configuration, some can be used as test and validation datasets and the others can be used for training.