How Good Are Deep Generative Models for Solving Inverse Problems?

Given the input $\mathbf{x}$ and a forward problem $\mathcal{F}$, the goal of inverse problems is to find solutions $\mathbf{y}$ such that $\mathcal{F}(\mathbf{y})=\mathbf{x}$. When $\mathcal{F}$ is not injective, there can be multiple plausible solutions $\mathbf{y}$ for the same observed data $\mathbf{x}$, i.e., $\mathbf{y}\in\{\mathbf{y}^{\prime}|\mathcal{F}(\mathbf{y}^{\prime})=\mathbf{x}\}$. In light of this, deep generative models estimate $p(\mathbf{y}|\mathbf{x})$ to capture the distribution of all plausible solutions.

When $\mathcal{F}$ is additive Gaussian, one way to model it is with diffusion models [22, 9]. Specifically, diffusion models break down $\mathcal{F}$ into $T$ additive Gaussian steps that progressively add noise to the data according to a predefined variance schedule. If we denote the original data as $\mathbf{x}_{0}$, then each step of the forward process $q(\mathbf{x}_{1:T}|\mathbf{x}_{0})$ can be written as:

Here, $\beta_{1},\dots,\beta_{T}$ represent the variance schedule. To model the reverse process $p_{\theta}(\mathbf{x}_{0:T})$, diffusion models make an additional assumption, namely that the reverse process is a Markov chain with Gaussian transition kernels, starting at $p(\mathbf{x}_{T})=\mathcal{N}(\mathbf{x}_{T};0,\mathbf{I})$:

where $\mathbf{\mu}_{\theta}(\mathbf{x}_{t},t),\mathbf{\Sigma}_{\theta}(\mathbf{x}_{t},t)$ are the predicted mean and covariance at time step $t$. It’s worth noting that the reverse process is only truly the inverse of the forward process when there are an infinite number of time steps $T$; otherwise the Gaussianity assumption in the transition kernel of the reverse process would not hold. In practice, a finite $T$ is used, in which case the Gaussianity assumption would introduce approximation errors. Therefore, the assumptions of diffusion models are technically not met when the forward problem is not additive Gaussian or when the number of time steps is small.

One alternative is to use a conditional GAN, comprising a generator and a discriminator. The generator $G_{\theta}$ takes in the observed input $\mathbf{x}$ and a latent code $\mathbf{z}\sim\mathcal{N}(0,\mathbf{I})$ and generates an output $\mathbf{y}\coloneqq G_{\theta}(\mathbf{x},\mathbf{z})$. The discriminator’s role is to distinguish between the generated sample and real data example. However, due to mode collapse, the generator tends to generate identical samples for the same input and ignores the latent noise [12]. To mitigate this problem, various approaches have been proposed, including introducing additional losses in the latent space [28], adding mode-seeking terms [19], or using contrastive losses [18, 25, 11]. While these methods succeed in enhancing sample diversity, they often come at the expense of reduced sample fidelity.

Another alternative approach is Implicit Maximum Likelihood Estimation (IMLE) [16]. IMLE differs from GANs in two key ways: it explicitly aims to cover all modes, and optimizes a non-adversarial objective. To achieve the former, IMLE reverses the direction in which generated samples are matched to real data: rather than making each generated sample similar to some real data point, it makes sure each real data point has a similar generated sample. To achieve the latter, it removes the discriminator (which matches generated samples to real data implicitly) and instead explicitly performs matching using nearest neighbour search. The latter can be done efficiently using DCI [14, 15], which avoids the curse of dimensionality. Recent advancements to IMLE include cIMLE [17], which extends IMLE to the conditional setting, and CHIMLE [20], which further enhances generated image quality by adopting a hierarchical approach to constructing latent codes.

A popular and statistically rigorous approach to perform uncertainty quantification is conformal prediction [4, 1, 7, 24]. Conformal prediction stands out for its ability to construct statistically valid uncertainty guarantees for general predictors without relying on distributional or model assumptions. Recent studies [2, 23, 10] have extended the application of conformal prediction to address image-to-image translation problems. In this study, we adopt a sampling-based approach, as outlined in [10], to evaluate model uncertainty across different methods.

The first evaluation focuses on assessing the validity of the model output as a solution to the inverse problem. Recall that in an inverse problem, it is essential that the provided input is similar to the solution to the forward problem with the model output as input. To assess this, we generate 50 random samples for each input image in the test set. These samples are then used as inputs to solve the corresponding forward problem for each task, namely bicubic downsampling by a factor of $16\times$ for super-resolution, conversion of RGB images to grayscale for colourization, and compression of images using JPEG at a quality of 1% for image decompression. Subsequently, we compare the processed outputs to the original input.

Table 1 shows the average LPIPS [26], PSNR and SSIM metrics between the model input and the solution to the forward problem applied to the model output. Lower LPIPS distances are better, whereas higher PSNR and SSIM are better. We found that CHIMLE [20], an IMLE-based method, performs the best across tasks and metrics. In second place, we have cIMLE [17] according to LPIPS and SSIM on super-resolution and colourization, DDRM [13] according to PSNR on super-resolution, and MoNCE [25] according to PSNR on colourization and according to all metrics on decompression.

To evaluate model uncertainty for each method, we adopt a sampling-based approach mentioned in [10]. For each input, we randomly generate 50 samples from each model and calculate the confidence interval at each pixel. In our visualization, the confidence interval is superimposed as highlights on top of the dimmed mean generated sample. A wider confidence interval corresponds to greater model uncertainty, visually appeared as a brighter colour. To determine the size of the confidence interval, we subtract the lower quantile from the upper quantile for each pixel within the set of generated samples for the same input. In our evaluation, we set the lower quantile at the $5^{th}$ percentile and the upper quantile at the $95^{th}$ percentile. Additionally, when visualizing the confidence interval for a specific input, we calibrate the lower and upper bounds using the remaining images in the test set, with more details available in [10].

Figure 1 shows the model uncertainty comparison in the task of $16\times$ image super-resolution. As shown, BicycleGAN [27], MSGAN [19] and NDM [5] generate samples with significant variations. However, these methods also show substantial uncertainty in regions that are constrained significantly by the input and therefore should not be uncertain, e.g., the thick black strips in the wing’s center should be just solid black with no other possibilities. On the other hand, DDRM [13] generates blurry outputs and shows low uncertainty in regions where the input is ambiguous, where one would expect a higher degree of uncertainty, e.g., along the rim of the butterfly’s wing. DivCo [18], MoNCE [25], and cIMLE [17] show model uncertainties that align with the degree of ambiguity present in the input, but their outputs are low in visual fidelity. CHIMLE [20] achieves the overall best performance and produces realistic outputs and uncertainty estimates that accurately capture input ambiguity. Visual comparisons of model uncertainty for the other tasks, namely image colourization and image decompression, can be found in the appendix.