XPDNet for MRI Reconstruction: an application to the 2020 fastMRI challenge

Zaccharie Ramzi CEA (Neurospin & Cosmostat), Inria (Parietal) Jean-Luc Starck AIM, CEA, CNRS, Université Paris-Saclay, Université Paris Diderot, Sorbonne Paris Cité Philippe Ciuciu Neurospin, Inria (Parietal)

(November 2020)

1 Synopsis

We present a new neural network, the XPDNet, for MRI reconstruction from periodically under-sampled multi-coil data. We inform the design of this network by taking best practices from MRI reconstruction and computer vision. We show that this network can achieve state-of-the-art reconstruction results, as shown by its ranking of second in the fastMRI 2020 challenge.

2 Main findings

We introduce the XPDNet, a neural network for MRI reconstruction, which ranked second in the fastMRI 2020 challenge.

3 Introduction

The acceleration of the scan time of Magnetic Resonance Imaging (MRI) can be done using Compressed Sensing (CS) [1] and Parallel Imaging (PI) [2, 3]. To have access to the underlying anatomical object $x\in\mathbb{C}^{n\times n}$ , one must solve the following idealized inverse problem:

M_{\Omega}FS_{\ell}x=y_{\ell},\quad\forall\ell=1,\ldots,L

(1)

where $M_{\Omega}\in\{0,1\}^{m\times n}$ is a mask indicating which points in the Fourier space (also called k-space) are sampled, $S_{\ell}\in\mathbb{C}^{n\times n}$ is a sensitivity map for coil $\ell$ , $F$ is the 2D Fourier transform, and $y_{\ell}\in\mathbb{C}^{m}$ is the k-space measurement for coil $\ell$ . Problem (1) is typically solved with optimization algorithms that introduce a regularisation term to cope with its indeterminacy.

\underset{x\in\mathbb{C}^{n}}{{\color[rgb]{0,0,1}\operatorname{arg\,min}}}\sum_{\ell=1}^{L}\frac{1}{2}{\color[rgb]{0,0,1}\|}y_{\ell}-M_{\Omega}{\color[rgb]{0,0,1}S_{\ell}}x{\color[rgb]{0,0,1}\|_{2}^{2}}+{\color[rgb]{0,0,1}R(x)}

(2)

In Eq. (2), the terms in color reflect parts of this optimization that are, to some degree, handcrafted and could benefit from learning. In plain text, the places where learning could improve the quality and the speed of the reconstruction are:

•

the optimization algorithm hyper-parameters (symbolized by $\operatorname{arg\,min}$ ) like step sizes and momentum;
•

the sensitivity maps $(S_{\ell})_{\ell}$ , usually estimated using ESPIRiT [4];
•

the regularizer $R(x)$ , usually $\lambda\|\psi x\|_{1}$ where $\psi$ is the decomposition over some wavelet basis;
•

the data consistency term $\sum_{\ell=1}^{L}\frac{1}{2}\left\|y_{\ell}-M_{\Omega}S_{\ell}x\right\|_{2}^{2}$ , usually derived from an additive white Gaussian noise model.

XPDNet, the neural network introduced in this abstract, leverages all the room for learning to provide an efficient, i.e. fast and accurate, MRI reconstruction method.

4 Methods

Cross-domain networks.

The general intuition behind cross-domain networks is that we are going to alternate the correction between the image space and the measurements space. The key tool for that is the unrolling of optimisation algorithms introduced in [5]. An illustration of what cross-domain networks generally look like is provided in Fig. 1.

Unrolling the PDHG.

The XPDNet is a particular instance of cross-domain networks. It is inspired by the PDNet introduced in [6] by unrolling the Primal Dual Hybrid Gradient (PDHG) algorithm [7]. In particular, a main feature of the PDNet is its ability to learn the optimisation parameters using a buffer of iterates, here of size 5.

Image correction network.

The plain Convolutional Neural Network (CNN) is replaced by an Multi-scale Wavelet CNN (MWCNN) [8], but the code¹¹1github.com/zaccharieramzi/fastmri-reproducible-benchmark allows for it to be any denoiser, hence the presence of X in its name. We chose to use a smaller image correction network than that presented in the original paper [8], in order to afford more unrolled iterations in memory [9].

k-space.

In this challenge, since the data is multicoil, we did not use any k-space correction network which would be very demanding in terms of memory footprint. However, following the idea of [10], we introduced a refinement network for $(S_{\ell})_{\ell}$ , initially estimated from the lower frequencies of the retrospectively under-sampled coil measurements. This sensitivity maps refiner [10] is chosen to be a simple U-net [11].

We therefore have 25 unrolled iterations, an MWCNN that has twice as less filters in each scale, a sensitivity maps refiner smaller than that of [10] and no k-space correction network.

Training details.

The loss used for the network training was a compound loss introduced by [12], consisting of a weighted sum of an $L_{1}$ loss and the multi-scale SSIM (MSSIM) [13]. The optimizer was the Rectified ADAM (RAdam) [14] with default parameters²²2tensorflow.org/addons/api_docs/python/tfa/optimizers/RectifiedAdam. The training was carried for 100 epochs (batch size of 1) and separately for acceleration factors 4 and 8. The networks were then fine-tuned for each contrast for 10 epochs. On a single V100 GPU, the training lasted 1 week for each acceleration.

Data.

The network was trained on the brain part of the fastMRI dataset [15]. The training set consists of 4,469 volumes from 4 different contrasts: T1, T2, FLAIR and T1 with admissions of contrast agent (labelled T1POST). The validation was carried over 30 contrast-specific volumes from the validation set.

5 Results

Quantitative.

We used the PSNR and SSIM metrics to quantitatively compare the reconstructed magnitude image and the ground truth. They are given for each contrast and for the 2 acceleration factors in the Figs. 2- 3. Similar results are available on the public fastMRI leaderboard³³3fastmri.org/leaderboards, although generally slightly better. It is a bit difficult to consider these results when compared to only the zero-filled metrics, but these quantitative metrics do not accurately capture the performance of the GRAPPA algorithm [2]. However, at the time of submission, this approach ranks 2nd in the fastMRI leaderboards for the PSNR metric, and finished 2nd in the 4 $\times$ and 8 $\times$ tracks of the fastMRI 2020 brain reconstruction challenge [16].

Qualitative.

The visual inspection of the images reconstructed (available in Fig. 2) at acceleration factor 4 shows little to no visible difference with the ground truth original image. However, when increasing the acceleration factor to 8, we can see that smoothing starts to appear which leads to a loss of structure as can be seen in Fig. 3.

6 Conclusion and Discussion

We managed to gather insights from many different works on computer vision and MRI reconstruction to build a modular approach. Currently our solution XPDNet is among the best in PSNR and NMSE for both the multicoil knee and brain tracks at the acceleration factors 4 and 8. Furthermore, the modularity of the current architecture allows to use the newest denoising architectures when they become available. However, the fact that this approach fails to outperform the others on the SSIM metric is to be investigated in further work.

7 Figures

Refer to caption — Figure 1: General cross-domain networks architecture. Skip and residual connection are omitted for the sake of clarity. $y$ are the under-sampled measurements, in our case the k-space measurements, $\Omega$ is the under-sampling scheme, $F$ is the measurement operator, in our case the Fourier Transform, and $\hat{x}$ is the recovered solution.

References

[1] Michael Lustig, David Donoho and John M. Pauly “Sparse MRI: The Application of Compressed Sensing for Rapid MR Imaging Michael” In Magnetic Resonance in Medicine, 2007 DOI: 10.1002/mrm.21391
[2] Mark A Griswold et al. “Generalized Autocalibrating Partially Parallel Acquisitions (GRAPPA)” In Magnetic Resonance in Medicine 47, 2002, pp. 1202–1210 DOI: 10.1002/mrm.10171
[3] Klaas P. Pruessmann, Markus Weiger, Markus B. Scheidegger and Peter Boesiger “SENSE: Sensitivity encoding for fast MRI” In Magnetic Resonance in Medicine 42.5, 1999, pp. 952–962 DOI: 10.1002/(SICI)1522-2594(199911)42:5<952::AID-MRM16>3.0.CO;2-S
[4] Martin Uecker et al. “ESPIRiT-An Eigenvalue Approach to Autocalibrating Parallel MRI: Where SENSE meets GRAPPA HHS Public Access” In Magn Reson Med 71.3, 2014, pp. 990–1001 DOI: 10.1002/mrm.24751
[5] Karol Gregor and Yann Lecun “Learning Fast Approximations of Sparse Coding” In Proceedings of the 27th International Conference on Machine Learning, 2010
[6] Jonas Adler and Ozan Öktem “Learned Primal-Dual Reconstruction” In IEEE Transactions on Medical Imaging 37.6, 2018, pp. 1322–1332 DOI: 10.1109/TMI.2018.2799231
[7] Antonin Chambolle and Thomas Pock “A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging”, 2011, pp. 120–145 DOI: 10.1007/s10851-010-0251-1
[8] Pengju Liu et al. “Multi-level Wavelet-CNN for Image Restoration”, 2018 URL: https://arxiv.org/pdf/1805.07071.pdf
[9] Zaccharie Ramzi, Philippe Ciuciu and Jean Luc Starck “Benchmarking MRI reconstruction neural networks on large public datasets” In Applied Sciences (Switzerland) 10.5, 2020 DOI: 10.3390/app10051816
[10] Anuroop Sriram et al. “End-to-End Variational Networks for Accelerated MRI Reconstruction”, 2020
[11] Olaf Ronneberger, Philipp Fischer and Thomas Brox “U-Net: Convolutional Networks for Biomedical Image Segmentation” In International Conference on Medical image computing and computer-assisted intervention, 2015, pp. 234–241 URL: http://lmb.informatik.uni-freiburg.de/
[12] Nicola Pezzotti et al. “An Adaptive Intelligence Algorithm for Undersampled Knee MRI Reconstruction: Application to the 2019 fastMRI Challenge”, 2020
[13] Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh and Eero P Simoncelli “Image Quality Assessment: From Error Visibility to Structural Similarity” In IEEE TRANSACTIONS ON IMAGE PROCESSING 13.4, 2004 DOI: 10.1109/TIP.2003.819861
[14] Liyuan Liu et al. “On the Variance of the Adaptive Learning Rate and Beyond” In Proceedings of International Conference for Learning Representations, 2020, pp. 1–3
[15] Jure Zbontar et al. “fastMRI: An Open Dataset and Benchmarks for Accelerated MRI”, 2018 URL: https://arxiv.org/pdf/1811.08839.pdf
[16] Matthew J. Muckley et al. “State-of-the-art Machine Learning MRI Reconstruction in 2020: Results of the Second fastMRI Challenge”, 2020, pp. 1–16 URL: http://arxiv.org/abs/2012.06318

T1 PSNR: 41.56 SSIM: 0.9506	T2 PSNR: 40.68 SSIM: 0.9554	FLAIR PSNR: 39.60 SSIM: 0.9321	T1POST PSNR: 42.53 SSIM: 0.9683

T1 PSNR: 38.57 SSIM: 0.9348	T2 PSNR: 37.41 SSIM: 0.9404	FLAIR PSNR: 36.81 SSIM: 0.9086	T1POST PSNR: 38.90 SSIM: 0.9517