Exploring the Power of Generative Deep Learning for Image-to-Image Translation and MRI Reconstruction: A Cross-Domain Review

The Generative Adversarial Network (GAN) [23] has proven to be effective in several applications, such as I2I translation, image super-resolution, image reconstruction, and data augmentation. GANs are inspired by two-player zero-sum games in game theory and consist of two components: the generator $G$ and the discriminator $D$. The generator $G$ is designed to learn the mapping from a noise distribution $p_{z}(z)$ to the data space $p_{data}$, while the discriminator $D$ operates as a binary classifier that distinguishes between the generated fake sample $G(z)$ from the distribution $p_{g}$ and the real sample $\mathbf{x}$ from the distribution $p_{data}$. The loss function of the GAN model can be expressed as:

	$\displaystyle\min_{G}\max_{D}L(G,D)=\mathbb{E}_{x\sim p_{data}(\boldsymbol{x})}[\log D(\boldsymbol{x})]$		(1)
	$\displaystyle+\mathbb{E}_{z\sim p_{z}(\boldsymbol{z})}[\log(1-D(G(\boldsymbol{z}))]$

Where ${E}_{x\sim p_{data}}$ is the expected value over all sampled real data instances, ${E}_{z\sim p_{z}(\mathbf{z})}$ is the expected value over all generated fake instances $G(z)$. Training a GAN model involves simultaneously training the generator $G$ and discriminator $D$. Given a batch of noise samples ${z^{(1)},...,z^{(n)}}$ from the noise distribution $p_{z}(z)$ and a batch of data samples ${x^{(1)},...,x^{(n)}}$ from the data space $p_{data}$, the discriminator $D$ is updated using its stochastic gradient ascent:

$$\nabla_{\mathbf{W}_{D}}\frac{1}{n}\sum_{i=1}^{n}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log\left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$

(2)

After updating the discriminator $D$, we need to update the generator $G$ by:

$$\nabla_{\mathbf{W}_{G}}\frac{1}{n}\sum_{i=1}^{n}\log\left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)$$

(3)

Where $\mathbf{W}_{D}$ and $\mathbf{W}_{G}$ represents the trainable parameters from discriminator $D$ and generator $G$.

The Conditional Generative Adversarial Network (CGAN) [59] is a GAN variation incorporating a conditional element. In a CGAN, the generator $G$ and discriminator $D$ are conditioned on additional input data, such as an image or a label. This allows the CGAN to generate samples that are conditioned on specific characteristics, such as specific object classes in an image or specific attributes in a dataset. The objective function of a CGAN can be written as follows:

	$\displaystyle\min_{G}\max_{D}V(D,G)=\mathbb{E}_{\boldsymbol{x}\sim p_{\text{data }}(\boldsymbol{x})}[\log D(\boldsymbol{x}\mid\boldsymbol{y})]+$		(4)
	$\displaystyle\mathbb{E}_{\boldsymbol{z}\sim p_{z}(\boldsymbol{z})}[\log(1-D(G(\boldsymbol{z}\mid\boldsymbol{y})))]$

The condition $\boldsymbol{y}$ could be a label or an image, $z$ is the random noise, and $x$ represents the ground-truth image. The generator $G$ takes the condition $\boldsymbol{y}$ and the random noise $z$ as input, while the discriminator $D$ receives the data pairs ($\boldsymbol{y}$, $G(\boldsymbol{y}$, $z$)) and ($\boldsymbol{y}$, $x$) and then determines which is real and which is fake. Figure 1 illustrates the architecture of both GANs and CGANs.

CS-MRI (Compressed Sensing MRI) reconstruction is a method for acquiring MRI images with reduced data acquisition time, which can be particularly useful in applications where prolonged scanning times are not feasible, such as with patients who have difficulty remaining still for extended periods of time [34]. CS-MRI operates by sparsely sampling the k-space data, the frequency domain representation of the MRI image, i.e., acquiring a small number of samples. This produces an under-sampled k-space, which can then be reconstructed using advanced mathematical algorithms, such as sparse optimization algorithms or deep learning-based methods, to produce an image. Certain image features can be reconstructed from a sparse set of k-space samples, reducing the amount of data needed to obtain an acceptable image quality. This can lead to faster scan times, enhanced patient comfort, and cost savings.

Given a MR image $\mathbf{x}\in\mathbb{C}^{N_{1}\times N_{2}}$, we do a fully-sampled measurement $\mathbf{y}$ on k-space, using the Fourier transform $\mathcal{F}(\cdot)$:

$$\mathbf{\hat{y}}=\mathcal{F}\mathbf{x}+\sigma$$

(5)

Where $\sigma$ represents noise. Using compressed sensing, we can reconstruct $\mathbf{x}$ from undersampled k-space measurement $\mathbf{\hat{y}}$ due to the sparsity characteristic of the MR image in its frequency domain. So that:

$$\mathbf{\hat{y}}=\mathcal{F}\mathbf{x}+\sigma$$

(6)

if we use a binary sampling mask $\mathcal{M}\in\mathbb{C}^{N_{1}\times N_{2}}$ is a method for selecting a subset of k-space lines in order to accelerate the measurement, then:

$$\mathbf{\hat{y}}=\mathcal{M}\times\mathcal{F}\mathbf{x}+\sigma$$

(7)

The sampling mask and the Fourier transform can be combined as an encoding matrix $\Phi$, in order to reconstruct $\mathbf{x}$, we need to minimize the noise, so that:

$$\mathbf{\hat{x}}=\arg\!\min_{\mathbf{x}}\|\Phi\mathbf{x}-\mathbf{\hat{y}}\|^{2}_{2}$$

(8)

Instead of using compressed sensing, non-generative DL-based approaches, such as CNNs, can be formulated as an optimization problem:

$$\mathbf{\hat{x}}=\arg\!\min_{\mathbf{x}}\|\Phi\mathbf{x}-\mathbf{\hat{y}}\|^{2}_{2}+\lambda\|\mathbf{x}-f_{DL}(\mathbf{x}_{0}|\phi)\|^{2}_{2}$$

(9)

Where $\mathbf{x}_{0}$ represents the zero-filled reconstruction, $f_{DL}(\mathbf{x}_{0}|\phi)$ is the output image from the trained model, $\phi$ is the optimized parameters in the trained model. Figure 2 shows the description of CS-based MRI reconstruction.

However, CS-MRI reconstruction has two major limitations: it requires a significant amount of computational time, and it is heavily dependent on the choice of regularization parameters and transforms basis, which must be adjusted carefully [56].

Previous works have shown the effectiveness of CGAN for I2I translation tasks. One notable example is the Pix2Pix model [33], which utilizes a CGAN to produce remarkable results in various applications, such as transforming satellite views into street maps and generating shoes from border sketches. The generator in Pix2Pix has a U-Net-like architecture that allows it to perform style transfer based on paired training images while preserving the pixel-wise characteristics of the source image. This results in accurate I2I translations combining high-level semantic and detailed pixel information. Zhu et al. [111] introduced CycleGAN, which used unpaired images as training samples combined with a cycle-consistency loss, producing more excellent results on image art style transfer. Wang et al. [91] proposed a new CGAN, HDPix2Pix, for high-resolution image synthesis, which can generate photo-realistic images without the need for hand-crafted losses or pre-trained networks. Almahairi et al. [2] then developed Augmental CycleGAN, which enables many-to-many mapping using unpaired data. Huang et al. [31] proposed MUNIT, a framework based on unsupervised I2I translation. MUNIT has outperformed CycleGAN in various I2I tasks. However, these related works to Pix2Pix have limitations in scalability and robustness when dealing with multiple domains. To address this, Choi et al. [13] proposed StarGAN, which can perform I2I translations for multiple domains using a single model. A ’domain’ is defined as images sharing the same attribute value, such as images of women or people with black hair. StarGAN v2 [14] is an updated version of the original StarGAN model [13]. StarGAN v2 uses the Adaptive Layer-Instance Normalization (AdaLIN) generator architecture, allowing the model to adjust the normalization parameters for each instance, improving performance for diverse domains. Additionally, StarGAN v2 has the capability to perform I2I translations in an unsupervised manner using unpaired training data. More recently, Cao et al. [9] introduced the Remix framework, which overcomes the over-fitting problem encountered with fewer training examples/paired images. The ReMix method can easily be incorporated into existing GAN models with minimal modifications. Table I shows some important GAN architectures for I2I translation.

Pix2Pix is a pioneering and groundbreaking I2I translation infrastructure. It is a large-scale, systematic I2I project that implemented supervised paired data training for two-domain image translation using an image-based conditional GAN. Pix2Pix employs U-net as the generator backbone to preserve low-level and high-level structural and textural information and PatchGAN as the discriminator capable of handling large images based on the fixed-size patch discriminator. Pix2Pix has been trained on a number of paired image datasets, including the Cityscapes dataset for semantic label vs. photo, the CMP Facades dataset for architectural label vs. photo, map vs. aerial photo from Google Map, and sketch and edge vs. photo. These models performed well during validation and testing and also provide an online resource for anyone to conduct their own trials.

General definitions: Pix2Pix uses an image-CGAN architecture that consists of two networks: a generator $G$ and a discriminator $D$. The aim of Pix2Pix is to translate an input image to a target image by learning a mapping and distribution based on a condition image. The generator network takes a conditional input image $\boldsymbol{x}$ and a random noise $\boldsymbol{z}$ as input. It generates a corresponding output image $\boldsymbol{\hat{y}}$, while the discriminator network assesses the realism of the generated image compared to the ground-truth image $\boldsymbol{y}$. Pix2Pix has been applied to various I2I translation tasks, such as converting maps to satellite images, sketches to photos, and gray-scale images to color images. Figure 3 demonstrates the details of the Pix2Pix architecture with an example of converting sketches to real objects. The objective and loss function of Pix2Pix is similar to those of a CGAN, which can be written as:

	$\displaystyle\mathcal{L}_{cGAN}(G,D)=$	$\displaystyle\mathbb{E}_{x,y}[\log D(x,y)]+$		(10)
		$\displaystyle\mathbb{E}_{x,z}[\log(1-D(x,G(x,z))],$

Additionally, as noted in previous research [67], the generator can sometimes produce images that are too dissimilar from the target images, making it easy for the discriminator to identify them as fake. To address this issue, including an $L_{1}$ or $L_{2}$ loss term in the generator’s loss function is common. This loss term measures the difference between the generated image and the target image and penalizes the generator for producing images that are significantly different from the target. As such, an $L_{1}$ or $L_{2}$ loss term should be added to the generator $G$ itself. In Pix2Pix, $L_{1}$ loss is used to prevent the blurring of images:

$$\mathcal{L}_{L1}(G)=\mathbb{E}_{x,y,z}\left[\|y-G(x,z)\|_{1}\right]$$

(11)

So that the final loss function to optimize the model is:

$$G^{*}=\arg\min_{G}\max_{D}\mathcal{L}_{cGAN}(G,D)+\lambda\mathcal{L}_{L1}(G)$$

(12)

U-Net generator: The Pix2Pix generator uses the U-Net architecture [71], which was state-of-the-art for image segmentation at the time of its development. The U-Net architecture consists of a series of down-sampling and up-sampling convolutional layers. The use of the U-Net architecture in the generator provides several benefits. One key advantage is its ability to handle large amounts of spatial information, making it well-suited for medical image segmentation tasks where the input images can be very large. Additionally, the symmetrical architecture of the U-Net enables the efficient transfer of information from the down-sampling path to the up-sampling path, preserving fine details of the segmentation mask. The U-Net also includes skip connections, which are implemented by concatenating activation from an earlier layer to the activation of a deeper layer. This allows the network to access high-level and low-level features, providing a complete representation of the input image. The skip connections also help capture long-range dependencies in the data by combining features from the contracting path with those from the expanding path.

PatchGAN discriminator: PatchGAN is a type of GAN discriminator that penalizes local image patch structures. It categorizes each $N\times N$ patch in an image as either real or fake. The size of the patch can be adjusted as needed. One benefit of the PatchGAN discriminator is its ability to apply a fixed-size patch discriminator on images of any size, unlike typical discriminators that only score the full image. Additionally, a smaller PatchGAN has fewer parameters, runs faster, and can be applied to images of any size.

Training and optimization: GAN models are notoriously challenging to train and converge due to the simultaneous training of the generator and discriminator, with improvements to one model often coming at the expense of the other. The differing learning rates of the two models can also make convergence difficult. As suggested in the original GAN paper [23], instead of training the generator $G$ to minimize $\log(1-D(x,G(x,z)))$, it is more effective to train $G$ to maximize $\log(D(x,G(x,z)))$ to improve convergence. In Pix2Pix, using instance normalization, which involves setting the batch size to 1, is a better choice than batch normalization for training generative models [87].

General definition: CycleGAN [111] is another well-known I2I translation benchmark. It extends Pix2Pix by enabling unpaired data domain transmission, thereby resolving the issue of insufficient data. CycleGAN’s objective, given two distinct image domains $X$ and $Y$, is to learn the mapping between the two domains so that an image from domain $X$ can be transformed into an image in domain $Y$ while preserving the original image’s content, and vice versa. It is used to enforce the cyclic consistency constraint between two image domains. If we translate an image from the domain $X$ to the domain $Y$ and then back to the domain $X$, we should obtain the original image. This constraint ensures that the generated images contain meaningful information and are not merely arbitrary noise. Figure 4 depicts the architecture of CycleGAN and its Cycle-consistency loss.

Loss function: There are two GANs in CycleGAN pipeline: $G_{X\rightarrow Y}$ represents the generator that map the image from domain $X$ to $Y$ and the discriminator $D_{Y}$ tries to identify the fake image $G_{X\rightarrow Y}(x)$ and image $y$ from domain $Y$; $G_{Y\rightarrow X}$ represents the generator that map the image from domain $Y$ to $X$ and the discriminator $D_{X}$ tries to identify the fake image $G_{Y\rightarrow X}(y)$ and image $x$ from domain $X$. So there are two adversarial losses; for example, from the generator:

	$\displaystyle\mathcal{L}_{\mathrm{GAN}}\left(G_{XY},D_{Y},X,Y\right)$	$\displaystyle=\mathbb{E}_{y\sim p_{\text{data }}(y)}\left[\log D_{Y}(y)\right]$		(13)
		$\displaystyle+\mathbb{E}_{x\sim p_{\text{data }}(x)}\left[\log\left(1-D_{Y}(G_{XY}(x))\right]\right.$

Meanwhile, the GAN that maps the image in domain $Y$ to $X$ also has an objective function:

	$\displaystyle\mathcal{L}_{\mathrm{GAN}}\left(G_{YX},D_{X},X,Y\right)$	$\displaystyle=\mathbb{E}_{y\sim p_{\text{data }}(x)}\left[\log D_{Y}(y)\right]$		(14)
		$\displaystyle+\mathbb{E}_{x\sim p_{\text{data }}(x)}\left[\log\left(1-D_{X}(G_{YX}(y))\right]\right.$

However, these two losses do not guarantee that the translated image will retain the domain-specific information. For instance, an image from domain $X$ that has been translated to domain $Y$ may share similar characteristics with images in the $Y$ domain while discarding all information from domain $X$. CycleGAN uses a specific loss function to let the networks keep domain information, which is called cycle-consistency loss. The idea of forwarded cycle-consistency loss is that for each image $x$ from domain $X$, the image translation cycle should be able to bring $x$ back to the original image, which means to minimize the distance between $x$ and $G_{Y\rightarrow X}(G_{X\rightarrow Y}(x))$. Similarly, the backward cycle-consistency loss is to minimize the distance between $y$ and $G_{X\rightarrow Y}(G_{Y\rightarrow X}(y))$. The cycle-consistency loss can be expressed by:

	$\displaystyle\mathcal{L}_{\text{cyc }}(G_{XY},G_{YX})$	$\displaystyle=\mathbb{E}_{x\sim p_{\text{data }}(x)}\left[\|G_{YX}(G_{XY}(x))-x\|_{1}\right]$		(15)
		$\displaystyle+\mathbb{E}_{y\sim p_{\text{data }}(y)}\left[\|G_{XY}(G_{YX}(y))-y\|_{1}\right]$

So that the total objective can be represented by:

	$\displaystyle\mathcal{L}\left(G_{XY},G_{YX},D_{X},D_{Y}\right)$	$\displaystyle=\mathcal{L}_{\mathrm{GAN}}\left(G_{XY},D_{Y},X,Y\right)$		(16)
		$\displaystyle+\mathcal{L}_{\mathrm{GAN}}\left(G_{YX},D_{X},Y,X\right)$
		$\displaystyle+\lambda\mathcal{L}_{\mathrm{cyc}}(G_{XY},G_{YX})$

CycleGAN is the first GAN-based unpaired I2I translation model, and it has made significant advances and has significant implications for future research. Importantly, the cycle-consistency loss has been widely used in future works [49, 96, 37, 22].

CNN is a potent DL model that has been widely applied to various computer vision problems ever since Alex et al. developed one of the most well-known CNN architectures, AlexNet [40]. CNN has evolved into a bench of permutations, such as VGGNet [79], ResNet [25], U-Net [71], etc. With the advent of GPUs, fairly deep CNNs such as ResNet have achieved remarkable success and can outperform most conventional techniques. However, CNNs are still considered black boxes at that time since they lack interpretation. Wang et al. [90] started to use CNN models instead of CS-MRI on the acceleration of MRI images by reducing the scanning time; they provide a supervised learning CNN model that can map the under-sampled images to fully-sampled images. Yang et al. [99] proposed a model-driven novel CNN model called ADMM-CSNet by reformulating the ADMM algorithms; it achieved better results than some data-driven CNN models.

After the CNN models become popular in the DL field, GANs can outperform conventional CNN models because they can learn data distributions significantly faster than CNN models. DAGAN [98] is the first well-known supervised GAN framework with conditions for MRI reconstruction. It employs a U-Net-like [71] architecture with the skip connection in the generator, a refined learning algorithm for fast convergence, and a novel combined loss function for superior reconstruction with preserved perceptual image details. Mardani et al. [57] provided a deep GAN to combine with the CS model, which can reduce the high-frequency noise and map zero-filled MR images to high-resolution MR images. Quan et al. [69] designed a dual-bench generator with ReconGAN and RefineGAN combining cyclic data consistency loss and generative adversarial loss, which can correctly map the under-sampled data to fully-reconstruction data. Li et al. [47] proposed the Structure-Enhanced GAN (SEGAN) with a new structure regularization called patch correlation regularization that aims to restore structure information locally and globally for efficient extraction of structure information. Murugesan et al. [60] proposed a novel GAN framework called reconstruction global-Local GAN (Recon-GLGAN), containing a generator and a context discriminator which incorporates global and local contextual information from images. Shaul et al. [77] introduced a practical, software-only GAN framework to estimate the missing k-space sample for brain MRI accelerating acquisition.

Most of the GAN models we discussed above use CNN-based generators and discriminators, especially U-Net and ResNet. They usually improve performance by redesigning loss functions or regularization. Recently, with the development of Vision Transformer (ViT), widely used in computer vision tasks that can outperform traditional CNN architectures, Some researchers have begun using ViTs as backbones instead of CNNs in GAN models. Korkmaz et al. [39] proposed a generative ViT called GVTrans, which can progressively map low-dimensional noise and latent variables onto MR images via cascaded blocks of cross-attention ViTs. Zhao et al. [109] demonstrated a GAN model called SwinGAN, which uses a Swin-Transformer-based backbone as the generator. Swin Transformer [50] is a U-Net shape hierarchical architecture ViT, which achieved robust results on various downstream tasks. Importantly, Lin et al. [48] given potential proof of the ViT-based GAN model can enable robust and fast accelerated MRI, using a pre-trained ViT model on natural images datasets, such as ImageNet, as the backbone of the GAN model that can be fine-tuned in MRI datasets which is costly to obtain. Table II shows some important GAN architectures for MRI reconstruction.

DAGAN [57] is one of the first de-aliasing and fast CS-MRI architectures that outperforms conventional CS-based methods and non-generative DL-based algorithms in MRI de-aliasing and reconstruction tasks. The DAGAN generator uses a CNN architecture similar to the U-Net [71] with skip connections, which is a highly advanced encoder-decoder network for medical image segmentation. DAGAN combines an adversarial loss with a novel content loss that considers both pixel-wise MSE and perceptual loss as defined by pre-trained VGGNet [79]. Additionally, frequency domain information from the CS-MRI is used as an additional data consistency constraint. Figure 5 illustrates the architecture of DAGAN.

In general, the DAGAN generator is based on the U-Net architecture and consists of 8 convolutional layers for down-sampling and eight corresponding convolutional layers for up-sampling. Each convolutional layer is followed by batch normalization and leaky ReLU layers, and there are skip connections between different levels of the encoder-decoder pairs. The discriminator performs a binary classification task between the generated de-aliased reconstruction image and the ground-truth image. The loss function of DAGAN is based on the GAN training process and can be expressed as:

	$\displaystyle\min_{G}\max_{D}L(G,D)=\mathbb{E}_{x_{t}\sim p_{train}(x_{t})}(\log D(x_{t}))$		(17)
	$\displaystyle+\mathbb{E}_{x_{u}\sim p_{g}(x_{u})}(-\log(D(G(x_{u})))$

Where $x_{t}$ represents the ground-truth images from fully-sampled k-space and $x_{u}$ represents the zero-filled image. The generator $G$ produces the corresponding de-aliased reconstruction $G(x_{u})=\hat{x}_{u}$, which is then evaluated by the discriminator $D$. A combined loss was designed for the generator’s training to improve the visual quality. This loss includes a pixel-wise mean square error (iMSE) loss, a frequency domain mean square error (fMSE) loss, and a perceptual VGG loss. The dual-MSE loss accounts for both the pixel-wise and frequency domain data, providing effective constraints, while the VGG loss enhances image quality. The two MSE losses for the image domain and frequency domain can be expressed as:

		$\displaystyle\min_{G}\mathcal{L}_{\mathrm{iMSE}}\left(G\right)=\frac{1}{2}\left\|\mathrm{x}_{t}-\hat{\mathrm{x}}_{u}\right\|_{2}^{2}$		(18)
		$\displaystyle\min_{G}\mathcal{L}_{\mathrm{fMSE}}\left(G\right)=\frac{1}{2}\left\|\mathrm{y}_{t}-\hat{\mathrm{y}}_{u}\right\|_{2}^{2}$

Where $\mathrm{\hat{y}_{u}}$ and $\mathrm{y_{t}}$ are the corresponding frequency domain data of $\mathrm{\hat{x}_{u}}$ and $\mathrm{x_{t}}$ after the Fourier transform. The VGG perceptual loss is represented by:

$$\min_{G}\mathcal{L}_{\mathrm{VGG}}\left(G\right)=\frac{1}{2}\left\|\mathrm{vgg}\left(\mathrm{x}_{t}\right)-\mathrm{vgg}\left(\hat{\mathrm{x}}_{u}\right)\right\|_{2}^{2}.$$

(19)

So, the total combined loss can be written by:

$$\mathcal{L}_{\text{TOTAL }}=\alpha\mathcal{L}_{\mathrm{iMSE}}+\beta\mathcal{L}_{\mathrm{fMSE}}+\gamma\mathcal{L}_{\mathrm{VGG}}+\mathcal{L}_{\mathrm{GEN}}.$$

(20)

The DAGAN [57] loss function for its generator consists of a combination of adversarial loss, pixel-wise mean square error (iMSE) loss, frequency domain mean square error (fMSE) loss and a perceptual VGG loss [79]. This combination of losses allows DAGAN to overcome smooth reconstruction difficulties and gain more anatomical information than traditional CNN-based MRI reconstruction algorithms. The adversarial loss is denoted as $\mathcal{L}_{\mathrm{GEN}}$, and $\alpha,\beta,$ and $\gamma$ are hyper-parameters that control the weight of each loss. The dual-MSE and VGG loss have been widely adopted in many subsequent works [35, 102, 55, 5].

Quan et al. [69] developed a highly regarded GAN-based MRI reconstruction framework, which blends a deep convolutional auto-encoder with the GAN architecture. This framework enables the utilization of deeper generator and discriminator networks to achieve improved performance. The framework is further enhanced by incorporating cyclic data consistency loss, which ensures faithful interpolation of under-sampled k-space data. Figure 6 shows the details of the Recon/RefineGAN framework.

The generator $G$ consists of two parts: ReconGAN and RefineGAN, each with an auto-encoder architecture with four encoder and decoder layers. Like DAGAN, Recon/RefineGAN proposed other loss constraints besides a general adversarial loss. They designed a cyclic data consistency loss $L_{cyc}$, which is a combination of under-sampled frequency loss $L_{f}$ and fully reconstructed image loss $L_{img}$ in a cyclic fashion. The cyclic data consistency loss can be represented by:

	$\displaystyle L_{cyc}(G)$	$\displaystyle=L_{f}(G)+L_{\text{img }}(G)$		(21)
		$\displaystyle=\mathbf{d}(m[i],\bar{m}[i])+\mathbf{d}(s[j],\bar{s}[j])$

Where $m[i]$ is the observed under-sampled data from the frequency domain, $\bar{m}[i]$ represents the data in the frequency domain after the $m[i]$ is taken into generator $G$ to get a reconstructed image, then apply under-sampled operator $RF$ (Fourier transform and sampling mask) on it. Similarly, $s[j]$ is the fully reconstructed image; if we use under-sampling operation $RF$ on $s[j]$, then we get $m[j]$ in the frequency domain that can be transformed into zero-filled MR image $s_{0}[j]$, the $\bar{s}[j]$ represents the image after $s_{0}[j]$ is taken into generator $G$. $\mathbf{d}$ represents distance metrics, such as mean-square-error (MSE), and mean-absolute-error (MAE).

The first deep learning model for multi-contrast CS-MRI reconstruction with fewer model parameters was proposed by Sun et al. [83]. Olut et al. [62] developed a GAN-based model with a new loss term that accurately captures the directional features of vascular structures and can reconstruct MRA images from T1- and T2-weighted MRI images. Song et al. [81] utilized a coupled dictionary learning method to reconstruct T1- and T2-weighted MR images simultaneously. Wei et al. [93] created a dual-domain GAN for multi-contrast MRI reconstruction using two pairs of generators and discriminators in both the image and frequency (k-space) domains. Dar et al. [17] proposed a CGAN-based [59] generative model, which improved the quality and versatility of multi-contrast MRI exams compared to traditional nonlinear regression and deterministic neural networks. It successfully extends the natural image synthesis task into the medical imaging domain. Dar et al. [18] developed a new method that builds upon their previous work [17] and addresses the limitations of pure learning-based reconstruction or synthesis by incorporating three important priors. Yurt et al. [103] created a multi-stream GAN architecture for multi-contrast MRI synthesis called MustGAN, which built the adaptive fusion of unique features in multiple one-to-one streams and shared features in a many-to-one stream in order to learn more latent representations and improve the synthesis ability. Do et al. [19] proposed a dual-GAN-based network to jointly reconstruct T1- and T2-weighted MR images, with the optimization of sampling patterns and training processes verified.

In Section II, we introduced the concept of conditional Generative Adversarial Network (cGAN). This technique is commonly used for image-to-image translation and synthesis tasks, such as converting grayscale images to color images and generating full images from partial images. In medical image synthesis, one of the most significant applications is the synthesis of T1- and T2-weighted MR images. Dar et al. [17] proposed a cGAN-based pipeline for the mutual translation of T1- and T2-weighted MR images. Instead of using the original adversarial loss function defined in equation (4), they utilized a squared adversarial loss function in their approach.

	$\displaystyle L_{\text{c}GAN}(D,G)=-E_{x}\left[(D(x\mid y)-1)^{2}\right]$		(22)
	$\displaystyle-E_{z}\left[D(G(z\mid y))^{2}\right],$

Where $x$ represents the ground-truth image, $y$ represents the source image (T1- or T2-weighted) used as a condition, $z$ represents the random noise, and $\hat{x}$ represents the reconstructed image. The cGAN has two branches of GAN pipelines that can translate between T1- and T2-weighted images.

In this section, we introduce a novel Double-Domain GAN (DDGAN) framework purposed by Wei et al. [93], which can reconstruct under-sampled T2-weight and MRA images from fully sampled T1-weighted images in a supervised manner. GAN has been applied in cross-modality image synthesis in recent years. In medical imaging, GAN-based image synthesis models can translate MRI, CT, and PET mutually. DDGAN is inspired by the possible combination of multi-contrast MRI reconstruction with synthesis. Specifically, DDGAN uses fully-sampled T1 k-space data as input with under-sampled k-space data from T2-weighted and MRA, which can share mutual information that improve the reconstruction quality of T2-weighted and MRA images. Moreover, DDGAN was created using a dual-domain GAN architecture, which consists of k-space domain GAN and image domain GAN. The k-space generator (K-generator) encapsulates the data using max-pooling and retains larger feature maps, which are more appropriate for data completion. A classical deep encoding network (U-net) is implemented for the image generator (I-generator) to extract multi-scale and dimensional features in image space for image denoising and enhancement [32, 80]. Together, these two GANs improved the efficacy of multi-contrast MRI reconstruction.

General definition: To illustrate it clearly, we can use the function to show the target for DDGAN:

$$\arg\!\min_{\mathbf{\theta}}L\{\textbf{DDGAN}(k_{u,T2},k_{u,MRA},k_{T1}|\theta),I_{T2},I_{MRA}\}$$

(23)

Where $L(\cdot)$ represents the loss function, $k_{u,T2}$ and $k_{u,MRA}$ represent under-sampled k-space data of T2-weight and MRA images, while $k_{T1}$ is the fully sampled k-space data of T1-weight images, $I_{T2}$ and $I_{MRA}$ are fully sampled T2-weighted and MRA images in the image domain. The modal DDGAN can generate reconstructed T2-weighted and MRA images which should be indistinguishable from $I_{T2}$ and $I_{MRA}$ from the image space discriminator. In general, the pipeline of DDGAN consists of two modules: the data completion module and the image enhancement module. Figure 7 shows details of DDGAN architecture.

Data completion module: In this module, a specific GAN with the generator and discriminator in the k-space domain are represented by $G_{k}$ and $D_{k}$, respectively. Through training, $G_{k}$ is able to learn a mapping from under-sampled to fully-sampled k-space data, which is then distinguished from ground truth by $D_{k}$. The input of $G_{k}$ includes the concatenation of fully-sampled k-space data in the T1-weighted image, under-sampled k-space data from the T2-weighted image, and MRA image. The output of $G_{k}$ includes a concatenation of fully-sampled k-space T1-weighted image, reconstructed k-space T2-weighted image, and MRA image, the $D_{k}$ should distinguish the $D_{k}$ output concatenation image from the ground-truth image (fully-sampled k-space T1-weighted image, T2-weighted image, and MRA image). $C$ is a data consistency operation that can replace the generated k-spaces with the data actually sampled. For GAN in the k-space domain, it provides a pixel-wised $L_{2}$-loss between generated k-space pairs and standard k-space pairs. The $L_{2}$-loss function is expressed by:

	$\displaystyle\arg\!\min_{G_{k}}L_{2}=\parallel C\odot G_{k}(k_{u,T2}\oplus k_{u,MRA}\oplus k_{T1})$		(24)
	$\displaystyle-(k_{T2}\oplus k_{MRA}\oplus kT1)\parallel_{2}^{2}$

Where, $C\odot$ is the data consistency operation, $\oplus$ is concatenation operation, $\parallel\cdot\parallel_{2}^{2}$ is the $L_{2}$ loss. The adversarial loss function in the k-space domain can be represented by:

	$\displaystyle\min_{G_{k}}\max_{D_{k}}L(G_{k},D_{k})=\mathbb{E}[\log D_{k}(k_{\text{T2}},k_{\text{MRA}},\boldsymbol{k_{\text{T1}}})]$		(25)
	$\displaystyle+\mathbb{E}[\log(1-D_{k}(G_{k}(k_{\text{u,T2}},k_{\text{u,MRA}},\boldsymbol{k_{\text{T1}}}))]$

Where $\boldsymbol{k_{\text{T1}}}$ is the given fully-sampled T1 image from k-space that can share information with other modalities. In general, the data completion module GAN can reconstruct k-space data from the under-sampled one, which can fill up the missing part of the k-space data.

Image enhancement module: In this module, the generator $G_{I}$ is taken inputs from an inverse Fourier Transform of the concatenation output of $G_{k}$ in the data completion module, including which means that $G_{I}$ can learn from fully-reconstructed k-space when the GAN in data complete module is finished training so that the GAN in image domain can focus on image enhancement and de-noising, instead of data translation from k-space. The output of $G_{I}$ can generate reconstructed T1-weighted MR images, T2-weighted MR images, and MRA images. The loss function in the image enhancement GAN can be represented by:

	$\displaystyle\min_{G_{I}}\max_{D_{I}}L(G_{I},D_{I})=\mathbb{E}[\log D_{I}(I_{\text{T2}},I_{\text{MRA}},\boldsymbol{I_{\text{T1}}})]$		(26)
	$\displaystyle+\mathbb{E}[\log(1-D_{I}(G_{I}(I_{\text{Gk,T2}},I_{\text{Gk,MRA}},\boldsymbol{I_{\text{Gk,T1}}}))]$

Experiments and validation of DDGAN can be divided into three distinct levels. First, it demonstrated the efficacy of dual-GAN architecture by comparing the performance of the k-space generator to that of dual-GAN. (k-space and image-space GAN). Second, it indicates that the completely sampled T1-weighted image is useful for reconstructing T2-weighted and MRA images, which is superior to reconstructing T2-weighted and MRA images using their k-space data alone. Comparing DDGAN to other GAN architectures, such as Pix2Pix, and CS-based MRI reconstruction algorithms revealed that DDGAN outperformed all baselines. In Section VI, experimental details will be provided.

For (I2I) translation tasks, several evaluation metrics are commonly used to measure the quality and similarity between the generated and ground truth images. Some examples of these metrics include:

Mean Absolute Error (MAE): MAE computes the average absolute difference between the image pixel values of the generated and ground truth images. It can be stated as follows:

$$\text{MAE}=\frac{1}{n}\sum_{i=1}^{n}\left|y_{i}-y_{i}^{\prime}\right|$$

(27)

where $n$ is the number of samples, $y_{i}$ represents the predicted value for the $i^{th}$ sample and $y_{i}^{\prime}$ is the actual value. The absolute difference between these values is calculated using the absolute value operator $\left|\cdot\right|$.

Mean Squared Error (MSE): MSE is the average squared difference between the image pixel values generated and the ground truth. The MSE is formulated as follows:

$$\text{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-y_{i}^{\prime})^{2}$$

(28)

The MSE represents the average squared difference between predicted and actual values. In contrast to the MAE, the MSE emphasizes larger differences and is sensitive to outliers. However, MSE and MAE are pixel-level metrics that can only determine the pixel-wise distance; they are hard to measure the structural and textural similarity.

FCN score [52]: is an evaluation metric used to quantify the performance of a semantic segmentation model. In particular, people use FCN scores for $\text{semantic maps}\leftrightarrow{\text{real photos}}$ tasks in I2I translation.

Structural Similarity Index (SSIM) [92]: SSIM takes into account changes in the mean, standard deviation, and covariance of the pixel values to determine the structural similarity between the generated and ground truth images. SSIM can be described as:

$$\text{SSIM}(x,y)=\frac{(2\mu_{x}\mu_{y}+c_{1})(2\sigma_{xy}+c_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+c_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+c_{2})}$$

(29)

The SSIM index evaluates the structural similarity between two images, $x$ and $y$, by examining their mean intensity values ($\mu_{x}$ and $\mu_{y}$), the variances of their intensity values ($\sigma_{x}^{2}$ and $\sigma_{y}^{2}$), and the covariance between the intensity values of the two images ($\sigma_{xy}$). To prevent instability in the division, $c_{1}$ and $c_{2}$ are utilized as small constants. The SSIM index ranges from -1 to 1, with a score closer to 1 indicating a high degree of structural similarity between the two images.

Peak Signal-to-Noise Ratio (PSNR): The PSNR is the ratio between the highest possible pixel value and the mean squared difference between the created image and the ground truth image. PSNR is represented by:

$$\text{PSNR}=10\log_{10}\left(\frac{\text{MAX}^{2}}{\text{MSE}}\right)$$

(30)

Where MAX is the maximum possible pixel value, MSE is the Mean Squared Error between the predicted and ground truth image. To assess image reconstruction quality, the PSNR compares the maximum possible pixel value with the mean squared error between the predicted and ground truth images. The higher the PSNR, the lower the reconstruction error and the higher the quality of the reconstructed image.

Fréchet Inception Distance (FID): In MRI reconstruction, The FID is a widespread metric used to evaluate the quality of generated images compared to real images. It is defined as:

$$\operatorname{FID}(P_{r},P_{g})=\left|\mu_{r}-\mu_{g}\right|{2}^{2}+\operatorname{Tr}\left(\Sigma_{r}+\Sigma_{g}-2\left(\Sigma_{r}\Sigma_{g}\right)^{1/2}\right)$$

(31)

Where $P_{r}$ and $P_{g}$ are probability distributions of real and generated images, respectively. $\mu_{r}$ and $\mu_{g}$ are the mean vectors of the real and generated image features, and $\Sigma_{r}$ and $\Sigma_{g}$ are their covariance matrices.

LPIPS (Learned Perceptual Image Patch Similarity) [107], which assesses the variety of the translated images and correlates well with a human perceptual similarity. It is calculated as the average LPIPS distance between two sets of translation outputs chosen randomly from the same input. A translated outcome that is more realistic and diverse will have a higher LPIPS score. The function of LPIPS can be expressed by:

$$d(I_{1},I_{2})=\frac{1}{W\cdot H}\sum_{x=1}^{W}\sum_{y=1}^{H}\left|f(I_{1}(x,y))-f(I_{2}(x,y))\right|_{2}$$

(32)

where $d(I_{1},I_{2})$ is the LPIPS distance between two images $I_{1}$ and $I_{2}$, $W$ and $H$ are the width and height of the images, respectively, and $f(x)$ is the deep neural network used to extract features from the images. The term $\left|f(I_{1}(x,y))-f(I_{2}(x,y))\right|_{2}$ is the Euclidean distance between the feature vectors extracted from the corresponding pixels in the two images. The final result is the average distance over all the pixels in the images.

Table 3 shows some important datasets that are used for I2I translation. Table 4 shows some datasets for MRI reconstruction, including T1, T2, PD, and MRA modalities for heart, knees, and brain research.

In the field of natural I2I translation, Pix2Pix has overcome several challenges and problems faced by researchers. Its most significant contribution is integrating separate, special-purpose machinery methods into a single framework that can solve different problems using a single pipeline [111]. Pix2Pix uses a U-Net instead of an auto-encoder as the generator’s backbone, which allows it to avoid losing structural information and preserve more details and textures. Additionally, Pix2Pix’s open-source software allows a wide range of tasks to be demonstrated without needing specific parameter tuning. Pix2Pix has become a successful benchmark in the I2I translation field, as evidenced by its use by various individuals, including artists, for their own purposes. However, Pix2Pix has limitations, such as the need for high-quality paired image data for training, which restricts the model’s ability to create diverse outputs. To address this, the same research group as Pix2Pix, Zhu et al., designed CycleGANs for I2I translation without using paired data as training samples, as the generators are trained using only unpaired data from both domains [111]. As a result, CycleGAN is a versatile tool for I2I translation when paired data is difficult or impossible to obtain. However, CycleGANs cannot fully bridge the gap between paired and unpaired data models, which means that the performance of models that use paired data is usually better than models using unpaired data. Researchers try to solve this by using weak or semi-supervised data, which could lead to a more effective solution, although labeling data is a time-consuming process. For instance, Park et al. achieved better results in I2I translation using unpaired data through contrastive learning and collaboration with the encoder-decoder, based on InfoNCE loss [63, 65].

Furthermore, Pix2Pix and its improved frameworks have limitations in generating multiple outputs or across multiple domains. To overcome this, Choi et al. introduced StarGAN v1 and v2 [13, 14], effectively addressing the multi-domain I2I translation problem. StarGAN uses a single pair of generators and discriminators to learn all mappings and generate multi-domain images. He et al. proposed the deliberation learning framework for I2I translation, which reviews and refines output images, building upon CycleGAN and StarGAN [26].

Preserving fine-grained details, such as texture and object boundaries, in the output image is often a challenge in I2I translation, and Pix2Pix also faces this limitation. To address this, HDPix2Pix was introduced, which can generate photo-realistic outputs by using a multi-scale generator network to handle complex structures in the output images [91]. Park et al. proposed the SPADE method, a conditional normalization method that provides better semantic information preservation than traditional normalization layers [66]. Lee et al. created DRIT++, a framework for disentangled representation that allows for preserving different image information and uses cross-cycle consistency loss for cyclic reconstruction training without paired data [43]. Shen et al. [78] claim that previous approaches, such as DRIT++ and Pix2PixHD, can handle complex structures in the output photos but may have serious limitations if the target domain images contain rich content and multiple disparate objects. They proposed an instance-aware approach that applies fine-grained local and global styles to the target picture spatially and improves the overall quality of the output through joint training.

In general, the challenges associated with I2I translation include the quality of the translation, the preservation of complex context and image structures, multi-domain I2I translation, and training with unpaired data. Researchers continue to enhance the performance of I2I translation tasks by refining algorithms and architectures, selecting suitable loss functions, integrating with sophisticated statistical approaches, and developing unique training pipelines.

Here, we highlight some of the key challenges researchers face in MRI reconstruction and suggest potential solutions based on a literature review. A major challenge is improving the perceptual quality of reconstructed images. Training networks solely on L1/L2 distance enforces overall image or patch similarity but fails to address local features such as anatomical sharpness, leading to blurring and fuzzy output. To address this, DAGAN [98] introduced new loss functions, such as the perceptual loss generated by a pre-trained VGG network, to extract high-level features from MRI images and enhance image quality and anatomical information. Chen et al. [10] proposed a novel encoder-decoder architecture that aims to increase reconstruction quality by preserving local features and sharpening anatomy by confining the GAN to follow the data distribution in the K-space, reducing the risk of artificial details. Li et al. [46] designed an edge-enhanced dual discriminator GAN that improves the reconstruction of edge details, preserving structural details and enhancing overall image quality. Higher-quality MRI reconstruction with fewer anti-aliasing artifacts may lead to more effective therapeutic applications.

Most MRI reconstruction algorithms reviewed rely on supervised learning and thus require fully-sampled ground-truth images, which can be time-consuming to acquire. One potential solution is to develop a semi-supervised or unsupervised learning model, although unsupervised learning often leads to subpar reconstruction quality. Bora et al. [7] proposed AmbientGAN, a statistical-based method for solving unsupervised image reconstruction using natural imaging datasets, opening up future MRI reconstruction research opportunities. Cole et al. [15] proposed a novel GAN method that quickly generates high-quality 3D MR images compared to baselines. When tested on a knee dataset, the gap between the proposed unsupervised GAN and the supervised GAN was relatively small, with a difference of 0.78% worse PSNR, 4.17% worse NRMSE, and equal SSIM. Efforts to reconstruct MR images without fully-sampled data may be faster and more accurate than conventional CS-MRI techniques. However, the state-of-the-art performance and results of supervised MRI reconstruction are still maintained.

Some studies have shifted their focus to image synthesis rather than reconstruction. For example, Sohail et al. [80] developed a GAN model to produce fake MR images across multiple modalities and contrasts. These fake images are newly generated signals rather than reconstructed ones, and thus the focus is solely on image synthesis and not reconstruction in the image domain. The ultimate goal of multi-contrast MRI reconstruction is to use the information from one fully-sampled contrast to reconstruct other MR modalities from the given sub-sampled k-space data. Similarly, [17] proposed a T1- and T2-weighted MR images mutual synthesis pipeline using only the image domain. Wei et al. [93] proposed a dual-domain GAN (DDGAN) that uses both fully sampled and subsampled k-space for optimal multi-contrast MRI reconstruction. Multiple contributions have been made by DDGAN, including the reconstruction of T2-weighted and MRA images using fully-sampled T1 and sub-sampled T2 and MRA k-space data. This demonstrates that thoroughly sampled T1-weighted k-space data can provide mutual information that facilitates the enhanced reconstruction of other modalities. Moreover, three-level experimental results indicate that DDGAN obtains the highest score across all evaluation metrics, demonstrating superior performance in recovering intricate structures and minimizing aliasing artifacts for accelerated multi-contrast MRI. But DDGAN has some problems, even though it is one of the newest and most advanced architectures for multi-contrast MRI reconstruction. The author has provided evaluation metrics that emphasize performance and precision, but they lack training performance. Was this DDGAN simple to train? Moreover, this architecture has incorporated fully sampled T1-weighted image pairs as a supervised term, which may be limited in terms of the data source. Finally, the author states that the I-generator requires larger feature maps for data completion, so using ViT-based architecture should be better than traditional CNNs because the self-attention layer in ViT makes it possible to embed information globally across the overall image [109, 20].