Fill in the          (a diffusion-based image inpainting pipeline)

Inpainting has been an important problem within the field of computer vision for many decades. It is a functionality that is essential to many image related applications such as object removal, image restoration, manipulation, re-targeting, compositing, and image-based rendering. It is also safe to believe that with the wide adoption of generative AI tools, functionalities like inpainting could see a significant increase in use within creative applications as well. As tools such as DALL-E, ChatGPT, Midjourney, and Sora are proving that they can be of high utility to those whose career depends on creative works, we are seeing for the first time how AI systems are significantly changing a large industry (The U.S. Bureau of Economic Analysis reports that arts and cultural production accounts for $1,016,249,142,000 and 4.4% of the U.S. economy, contributing 4,851,046 jobs  [4]. Therefore, improving functionalities such as inpainting and allowing them to have a more robust range of capabilities can have a true economic impact.

There are two primary general approaches to the problem of inpainting: stochastic and deterministic. Stochastic methods output multiple sensible inpainting results through a random sampling process, while deterministic methods produce a single result. A common deterministic approach is to take an image along with a binary mask representing the region to be inpainted and pass these two images into a generator model such as a GAN trained to fill in the missing region. The task of improving the generator’s ability to produce acceptable inpaintings has been tackled from a wide range of approaches including attention mechanisms, encoder-decoder connections, deep prior guidance, and multi-scale aggregation  [5]. Stochastic inpainting techniques, such as flow-based and MLM-based methods, utilize generative models and sequence prediction to reconstruct image structures and textures. Additionally, many stochastic techniques take the approach of starting with a noisy image and iteratively denoise it until reaching a sensible output, a strategy commonly used by diffusion-based inpainting techniques which we will focus on.

An in-depth overview of RePaint’s pipeline is necessary to understand our contribution. Initially, the diffusion process transforms the masked starting image $x_{0}$, into Gaussian white noise $x_{T}$ by adding iid Gaussian noise with progressively greater variance to a scene image. At each time step $t$ of the forward pass, the image undergoes transformations according to the diffusion process, becoming noisier as $t$ increases until it reaches pure noise at high $t$ values.

In the backward pass, the goal is to reconstruct the original image from the white Gaussian noise. At each time step $t$, the noisy image is inputted into a DDPM. The output of the masked-out portion of the image is extracted and combined with the non-masked region obtained from the corresponding forward pass step. This process gradually restores the original image by filling out the masked-out portion as accurately as possible. (see Figure 1 for the pipeline diagram of RePaint). RePaint also introduces jumping and sampling to improve results, which will be explained in Section 3.3.

Our contribution is introducing the ability to specify an object to be inpainted into the scene at a specific location. The pipeline is designed to take in three input images: the background scenery, the target image, and a binary mask representing the region of interest within the target image. Notably, unlike RePaint’s pipeline, which solely processed the scene alongside an arbitrary mask, our approach incorporates contextual information for more target inpainting. In addition to denoising the scene during the forward pass, our method also applies denoising to the target image using a similar procedure. During the backward pass, the image at timestep $t$, $x_{t}$, is passed to the DDPM as normal. This is described in equations 1-2, which were inspired by equations 8a-c in [3]:

where $\bar{\alpha}_{t}:=\Pi_{i=1}^{T}(\beta_{i})$.

While the unmasked region of the scene is extracted from the corresponding forward pass, the same as RePaint, a notable issue arises in handling the masked target, since we have both the masked noised object from the forward pass and the masked object from the DDPM. This discrepancy, which we call “mask conflict”, arises from the coexistence of two masked versions of the object: one generated during the forward pass and the other by the DDPM. To resolve this conflict and produce the image at $x_{t-1}$, we take a convex combination of the two masked target images using a parameter series $\lambda_{t}$, described as

Finally, to achieve our image $x_{t-1}$ in the backward step pass, we take the scene from the forward pass and the result of the ”mask conflict” using the binary mask,

where $m$ represents the binary mask. This entire process is described in Figure 2.

In the pipeline (Figure 2), each denoising (backward) step is a function of only the noised scene image, noised target image, and previous noised combination. During the linear combination to calculate the DDPM input, there exists the possibility that the border between the target area and masked, noised scene image has a sudden, unnatural change in color. The RePaint authors noted a similar issue with the combination of the previously noised combination and the masked scene and proposed resampling to address this issue. Resampling is a trick to increase the diversity and smoothness of the inpainted image by noising it and running it through the DDPM multiple times. This allows the DDPM to predict the pixels on both sides of the mask’s boundary which increases the quality of these regions (along with the added benefit of increasing the variance of pixel values generated inside the mask). For example, first, the linear combination of a noisy, masked scene and generated image is passed into the DDPM. The result of this operation is a slightly less noisy combination with jarring boundaries. Gaussian noise is added to this result which is then passed back into the DDPM. This process is repeated $r$ times, after which the boundary transition is smoother. Since running the DDPM is time and computationally-intensive, the RePaint authors propose a jumping schedule parameter $j$ to control when resampling occurs. Every $j$ timesteps of the backward pass, $r$ resampling steps occur. When $r<=j$, the pipeline has the same runtime and FLOPs as the original pipeline ($r,j=1$).

As discussed in Section 3.2, $\lambda_{t}$ controls the convex combination of the generated inpainted target with the noised ground truth target at timestep $t$. Note that with $\lambda_{t}=1$, our pipeline collapses into RePaint’s original pipeline since there would be no contribution from the noised target in the forward step target images. With this pipeline design, we had an initial hyperparameter search over various values for $\lambda_{t}\in[0.8,0.9,0.993,0.995,0.999,0.9999]$. Additionally, for the jump length and jump size variables, we tried values in the range $[10,20,30,40]$. Finally, for the timestep count, we tried values in the range $[50,100,150,200,250]$.

As expected our result ends up looking more like random noise or takes on an uninterpretable shape the closer our lambda value is to one. However, smaller lambda values result in an output that looks like the target image was simply cropped out and pasted on top of the scene background. We found that lambda values in the range of 0.92-0.97 performed the best at keeping the content of the target image without making it seem like it was an exact copy and paste. Additionally, we expected to see that as our sampling parameter ($t$) increases to a large value, our model’s output is less fuzzy/noisy. This makes sense since this would put less weight on our forward pass noised target images in the denoising steps.

Finally, we noticed that increasing our jump and jump length parameters allowed for all-around higher quality outputs. This makes sense since the whole purpose of jumping is to allow the model to generate output which takes into account the surrounding scene’s context as much as possible when having the DDPM generate output. In summary, we performed a standard grid search over the ranges of all these parameters and identified the combination of optimal values to be 40 for both jump length and jump size, 200 for timesteps, and 0.993 for $\lambda_{t}$.

During our experimentation, we identified two viable methods for defining the target mask. The first approach involved creating an exact cutout of the target object, allowing the model to generate a more detailed image. However, this method often resulted in an unnatural-looking transition from the scene to the target object, appearing as if the object was simply pasted onto the image. Alternatively, we explored a more lenient approach by including additional information surrounding the target object in the mask. While this led to a more natural-looking transition between the scene and the target object, the resulting image of the target tended to be less detailed and smooth. Another issue with this approach was the dependency on the target image being in a similar environment to the scene. For example, if the target was a dog standing on gravel and the scene was a grass field, it would not make sense to give the model the grey-colored gravel border to inpaint into the final image. To address this, we investigated alternative modifications to the exact mask approach, as outlined in Section 4.1.

The biggest issue we found from our results was the unnatural-looking boundaries in our unpainted images. Currently, with an exact binary mask, the pipeline didn’t have context about how ”close” a pixel was to the boundary. Thus, to help provide context and more flexibility to the DDPM in the boundary error, we decided to explore some alternative masking methods.

Additionaly, we explored increased integration with RePaint’s model to address issues of lacking detailed features and producing smooth images with precise masks or jarring boundaries. This involves relying less on the noised target from the forward pass and more on the previous DDPM generation. Specifically, for timesteps $t$ closer to 0 in 3, we set $\lambda_{t}=1$ so that the pipeline can infer and create a more natural border around the inpainted target portion for the remainder of the timesteps. While retaining the same input structure (scenery, target, mask target), we modified the backward pass to transition to RePaint’s model by scheduling $\lambda_{t}$ to be a linear interpolation 4 from 0 to 1 for timesteps $T$ to $pT$, such that $p\in\left[0,1\right]$ and 1 for all $t<pT$.

Based on our results from Section 3.4 (particularly note 5), we let the hyperparameters $r,j=40$, reduce $T$ to $100$, solely to generate samples quicker for experimentation and conduct a grid search for $p\in{0.1,0.25,0.5,0.75,0.9}$.

We found that as $p$ increases, the inpainted image becomes less faithful to the target and more blurry. This is because a high value of $p$ implies that the pipeline is denoising without any direct contribution from the target image for $pT$ denoising steps. For such denoising steps, the pipeline is equivalent to RePaint, so the generated images will have high variance as noted in Guided Diffusion [1] and RePaint [3].

We identify the best value for hyperparameter $p$ as $0.5$ for the inpainting task. This setting allows for the inpainted image to be similar to the target image while allowing the pipeline to seamlessly fill in the outer border of the target subject. Additionally, this value allows the pipeline to retain the semantic meaning from the target image while inpainting novel and semantically consistent additions (e.g., the bow added to the dog in 5(c) and the grass interacting with its paws)

As described previously, there is a high variance in the final inpainted when running with $p>0.5$. The case when $p=0.5$ is particularly interesting because much of the semantic information of the target image is kept but the high number of denoising passes through the DDPM in this scenario makes the generations creative.

For a run with $T=100,r,j=40,$ and $p=0.5$, a giraffe target image and grassland scene (middle right image in Figure 6), the resulting image maintains the shape and color of a giraffe due to the masking $\forall t$ denoising steps but the content inside of the mask resembles a flamingo-giraffe as if it came from a Truffula forest scene from Dr. Seuss’ The Lorax. After rerunning with these hyperparameters multiple times, we could not produce a result that was so distinct from a giraffe. To mitigate this failure mode in a production setting, we recommend producing at least two candidate images per scene and target.

Another failure mode when $p>0.5$ is that this setting reveals the biases of the DDPM train data. As the RePaint authors note, a DDPM model trained on ImageNet, as is the one used to generate all images in this report, will be biased towards denoising dogs [3]. For iterations $t$ of our pipeline such that $\forall\tau<t,\lambda_{\tau}=1$, the autoregressive nature of the denoising process means that all future steps in the denoising process are biased towards high likelihood images in the DDPM dataset.

To address this failure mode, we recommend using a model like ResNet [2] to find the class of the target image and using a DDPM trained on a dataset where that image class is not underrepresented.

The authors present Fill in the      , a diffusion-based inpainting pipeline that seamlessly inserts target images into scenes. Based on the current analyses, the authors recommend the following hyperparameters: diffusion timesteps $T=200$, jumping steps $j=40$, resampling steps $r=40$, and a piecewise linear $\lambda$ schedule with $p=0.5$ (i.e., $\lambda_{t}=1$ if $t\leq 0.5T$ and a linear interpolation from $(pT,1.0)$ to $(T,0)$ elsewhere.

Moving forward, there are several avenues for further exploration and improvement in our research. First, we can delve deeper into mask modifications, as described in Section 4.1. The majority of our experimentation was with hand-made exact binary masks, but exploring alternative masking techniques such as gradient-based or buffered masks may allow the model to produce more realistic-looking inpaintings while still keeping the details of the original target. Furthermore, while the introduction of lambda scheduling in Section 4.2 improved the performance of our core pipeline, the addition of a more dynamic-based lambda scheduling could further enhance the adaptability and versatility of the pipeline. This dynamic approach would involve adjusting the lambda values based on the characteristics of different scene-target pairs, therefore optimizing the inpainting process for each specific scenario.

In addition to exploring mask modifications and refining lambda scheduling, expanding our testing to encompass a wider variety of images would be a huge improvement. Currently, the manual creation of the target mask poses scalability challenges. A solution could be to leverage segmentation techniques that would automate mask generation, facilitating the creation and efficient testing on a larger dataset. With a larger dataset, it would be much easier to analyze a diverse range of scene-target pairs, leading to optimal hyperparameters that generalize well across various scenarios.

Finally, our research aims to achieve a fully automated pipeline for inpainting. This system would require minimal user input, consisting mainly of providing a scene and target images and selecting the inpainted region while automating the target position, mask creation, and choice of hyperparameters.

In this study, we aimed to improve image inpainting techniques by enhancing control over what exactly gets generated. In particular, we identified that current approaches did not allow a model to base its inpainted output on an image of a target object as opposed to textual prompting or other ways of specifying a desired inpainted object. Leveraging recent advancements in generative AI and inpainting methodologies, we focused on a diffusion-based approach where we modify the inputs given to a diffusion model on its denoising steps to feed it the target image’s context. Through some adjustments and refinements, we sought to mitigate challenges such as mask conflicts and boundary realism, achieving some interesting results in the quality and naturalness of inpainted images with target objects. Our experiments provided valuable insights into the effectiveness of our approach and the importance of various hyperparameters. Overall, our project represents a modest yet meaningful step forward in the field of image inpainting.

We would like to kindly thank the CSE 493G staff for the amazing lecture content and guidance that allowed us to complete this project. Additionally, we express our appreciation to the RePaint paper and its authors, which was a huge inspiration to our paper.