NeRFocus: Neural Radiance Field for 3D Synthetic Defocus

Neural Radiance Field (NeRF) [21] is one of the exciting pieces of work that has emerged in recent years at the intersection of neural networks and computer graphics. Previously, researchers have explored the potential of MLP to represent graphic properties implicitly [7], [31], [34], [10]. Volumetric representations [15], [25] and volume rendering [11], [18] also demonstrated the power of rendering high-quality novel views. Further, Mildenhall et al. proposed NeRF [21], which uses an MLP to represent a continuous volumetric scene function. By explicitly designing the view-dependent effects and using positional encoding, NeRF significantly improved the performance for rendering realistic complex scenes.

Methods has been proposed to further extend the performances of NeRF, e.g., accelerate training or inference time [23], [36], [9], [41], [16], improve the generalization performance [43], [38], [40], [4], enable dynamic scene rendering [29], [13], [42], use unstructured images for training [17], [27], extend NeRF for multiscale and unbounded secens [23], [2], [3], scene disentanglement and control [44], [26], [24], [28], etc. This paper proposes the first framework that extends NeRF to approximate thin lens imaging, so the defocus effects can be rendered.

Our method is directly inspired by mip-NeRF [2], a multiscale representation for anti-aliasing rendering. Mip-NeRF extends the pixel’s receptive field to a circle. Instead of sampling rays, mip-NeRF sampling conical frustums and use the Integrated Positional Encoding (IPE) to make the encoded frequency components reflect not only the changes of position but also the variance of frustum size. However, mip-NeRF’s modeling is based on pinhole imaging, thus can not render lens effects. We explicitly model the thin lens imaging and derive composite cones that can be used to render each pixel into equivalent lens effects.

In general, high-quality defocus effects heavily rely on expensive wide-aperture lenses. Thanks to advances in machine learning, some approaches have emerged to synthesize defocus effects from single or multiple large DOF images that mobile cameras can easily obtain. A simple solution for synthetic defocus is using a segmentation network [8], [5] to separate the foreground and background, then applying blur operation on the segmented background [32], [33], [39]. More general approaches are based on multiplane representations. For instance, one can apply depth estimation [14], [30] to divide the scene into several layers of different depths, then use depth-dependent kernels to blur these layers and composite them as an image to synthesize defocus effects [22], [45]. Similar methods can be applied to dual-lens images where the predicted disparity plays the role of depth [1], [39]. These image-based methods simplify the defocus effects in lens imaging as the “gather” operation on discretized depth layers, i.e., each pixel gathers influences from nearby pixels to form its color, which is usually implemented by convolution with a circular blur kernel. “Gather” operation may suffer edge artifacts if the depth or disparity estimation is inaccurate. Besides, the multiplane representation is hard to handle the occluded area between its discretized depth layers, making it difficult to synthesize free-view 3D defocus effects.

Recently, Mildenhall et al. [19] proposed RawNeRF and integrated it with the multiplane representation to synthesize 3D defocus effects. By training on raw images, RawNeRF can render images with a high dynamic range (HDR) that can be further retouched, such as changing exposure, tone mapping, and focus. To synthesize defocus effects, they first precompute a multiplane representation using the trained RawNeRF, then use the mentioned layer-wise blur operation to synthesize 3D defocus effects. However, NeRF-based precomputation of multiplane representation is either time-consuming or memory-consuming. Instead, NeRFocus, based on precise optical modeling, can directly render effects without post-process. Besides, our method should be compatible with RawNeRF’s “HDR training”, which may help improve our visual quality of defocus effects, especially the saliency of defocused bright highlights.

The neural radiance field (NeRF) [21] is a field of radiance (i.e., view-dependent color) and density that is consistently represented by a multilayer perceptron (MLP). To calculate the color of a pixel on the sensor plane, NeRF projects a ray r from the pixel to the pinhole and passes it through the scene space, then samples the ray by interval $\delta_{i}$, $i\in\{1,...,N\}$ to form $N$ sample points $\textbf{x}_{i}$. Positional Encoding (PE) is applied on r and each sample point $\textbf{x}_{i}$:

where $\textbf{x}_{i}$ represents the 3D location $(x_{i},y_{i},z_{i})$, $L$ and $M$ are hyperparameters, denoting the number of frequency components in PE. Tancik et al. [37] reveal that the MLP is insensitive to high-frequency variance unless explicitly provided, which makes PE essential to the success of NeRF. The resulting frequency components $\gamma(\textbf{x}_{i})$ and $\gamma(\textbf{r})$ will be fed into the MLP to predict the corresponding color $\textbf{c}_{i}$ and density $\sigma_{i}$:

The predicted colors and densities will be used for discrete volume rendering [18] to generate the pixel color $\textbf{C}(\textbf{r})$:

The same rendering process is applied on every pixel of the sensor plane and finally composite a complete image.

Given a sparse set of posed images for training, NeRF chooses a pose for rendering and compares the result with the ground truth (GT) corresponding to the camera pose. Owing to the differentiable volume rendering formulation and the continuity of predicted color $\textbf{c}_{i}$ and density $\sigma_{i}$, the gradient descent of loss function can smoothly backpropagate to the MLP parameters, thus making the MLP easy to converge to a plausible scene representation. In this way, NeRF achieves novel view synthesis without depth supervision.

The thin lens model ignores optical effects due to lens thickness and is usually used to simplify the analysis of lens imaging. The part (a) in Fig.2 illustrates a thin lens model. Given a thin lens with aperture diameter $A$, focal length $f$, and focus distance $l$, we can calculate the image distance $l^{\prime}$ following the Gaussian lens equation:

Note that all “distance” in this paper denotes the distance along the optical axis, i.e., the z-axis by our definition. The focal plane is at the focus distance $l$ and is orthogonal to the z-axis. The sensor plane is also orthogonal to the z-axis but located at the image distance $l^{\prime}$. Here we take a point $e$ in the scene space for analysis. If $e$ is away from the focal plane, its imaging on the sensor plane is a circle rather than a point. This circle is called the circle of confusion (CoC) and is the immediate cause for defocus effects. Given $e$ at distance $z_{e}$, we can get its image distance $z_{e}^{\prime}$ by Gaussian lens equation and use the properties of similar triangles to calculate its CoC diameter $d_{e}$:

If we move $e$ on the plane “z=$z_{e}$”, its CoC diameter $d_{e}$ is invariant. This attribute is critical for the following derivation.

Existing NeRFs are based on a pinhole imaging model, where the radiance of each object point in scene space is directly passed through the pinhole and imaging on the sensor plane as a point, without any lens-like optical refraction. Therefore, the whole scene is in sharp focus and defocus effects can not be rendered.

A typical way to approximate thin lens imaging in ray tracing is to sample enough rays and calculate the average returned radiance. But this will be computational expensive for NeRF, as the rendering of each ray needs dense predictions using MLP. Instead, we inverse the thin lens imaging to calculate the beam path of each pixel and directly render them. Fig.2 and Fig.3 illustrate our modeling.

Let’s put the point $e$ on the sensor plane for analysis. Following the same assumption in NeRF that the ray path is reversible, we project rays from $e$ to every point on the lens plane. Actually, the corresponding refracted beam path constitutes a bicone that can be described in closed form following the lens equation. Say we have a plane “z=$z_{\alpha}$” denoted as $\alpha$, the bicone’s cross-section on $\alpha$ should be a perfect circle, whose diameter $d_{e}$ can be calculated following Eq.5, but with different parameters (symmetrically, if take $e$ as the scene point and $\alpha$ to be the sensor plane, this circle can be seen as the CoC of $e$ on the plane $\alpha$):

Ideally, we can get the point $e$’s color by calculating the radiance returned by the bicone. But, how to get the color of a pixel? NeRF uses the color of a point to represent the color of a pixel, while in Mip-NeRF [2], Barron et al. extend the receptive field of a pixel as a circle, which achieves better rendering quality. Our method can easily incorporate this extension, as is shown in the (c) part of Fig.2. Say we have a pixel, and the point $e$ is located at the pixel center. We use a circle centered on $e$, with diameter as $d_{0}$, to represent the receptive field of the pixel. Follow the practice in mip-NeRF, we set $d_{0}$ to the width of the pixel scaled by $2/\sqrt{3}$ for all experiments. The pixel’s beam path forms a composite cone, which is overlapped by the beam paths of the points inside the receptive circle. We project lines from each point inside the circle through the lens center. Each line can be seen as the axis of a bicone. These lines constitute a cone (purple region in Fig.2). Based on parallel similarity, the cone’s cross-section on $\alpha$ plane should always be a perfect circle, with diameter:

Let’s take the cross-section on $\alpha$ plane for analysis. Since the cone’s cross-section is a circle with diameter $d_{c}$, and each bicone’s cross-section is a circle with diameter $d_{e}$, the bicones overlap as a composite cone, whose diameter on $\alpha$ plane is:

where we can see that four parameters fully characterize the composite cone: pixel’s receptive diameter $d_{0}$, aperture diameter $A$, focal length $f$, and focus distance $l$. Then we can get the pixel’s color by calculating the composite cone’s returned radiance. To simplify the computation, we partition the composite cone by putting the $\alpha$ plane on different distance intervals $z_{i}$, $i\in\{1,...,N+1\}$ to form $N$ conical frustums $\textbf{v}_{i}$. We want to calculate a radiance $\textbf{c}_{i}$ and density $\sigma_{i}$ to represent each frustum $\textbf{v}_{i}$, so we can use Eq.3 to calculate the returned radiance. But the problem is, how should we calculate the representative radiance and density?

In theory, every point inside the frustum may contribute radiance, but their “importance” may not equal, as the bicones in the composite cone overlap unevenly. Intuitively, a point close to the composite cone axis should contribute more because more bicone is overlapping. Therefore, we model the importance distribution inside frustum $\textbf{v}_{i}$ as a 3D Gaussian distribution $\mathcal{N}(\boldsymbol{\mu}_{i},\boldsymbol{\Sigma}_{i})$, with independent distribution of each coordinate $x,y,z$. Theoretically, our frustum’s cross-section is parallel to the sensor plane while mip-NeRF’s is orthogonal to the cone axis. But these differences bring negligible varies for the encoded value of frustum. So we adopt Mip-NeRF’s coordinate system and it’s IPE for simplicity. The mean value vector $\boldsymbol{\mu}_{i}$ is composed of $(\mu_{x},\mu_{y},\mu_{z})$, the same value as the frustum’s centroid coordinate. The covariance matrix $\boldsymbol{\Sigma}_{i}$ is a diagonal matrix with elements $\sigma_{x},\sigma_{y},\sigma_{z}$, i.e., the variances along each coordinate. These variances are set as the same value as the frustum’s uniform distribution variance along each axis.

Therefore, we define the representative radiance $\textbf{c}_{i}$ and density $\sigma_{i}$ of a frustum $\textbf{v}_{i}$ by calculating the expectation of radiance and density of every point inside this frustum:

A direct way to implement this operation is sampling enough points inside the frustum $\textbf{v}_{i}$ follow the 3D Gaussian, using an MLP to predict their radiance and density, then calculate the average radiance and density. But this is extremely time-consuming. Inspired by Mip-NeRF, we use an MLP to directly predict the radiance of frustum $\textbf{v}_{i}$, with the expected value of $\gamma(\textbf{x})$ as input.

Specifically, each dimension $x,y,z$ of the 3D Gaussian is independent of each other. Moreover, each frequency component in positional encoding (PE) is independent of each other as well, thus the distribution of each PE component can be simplified as a 1D Gaussian distribution. Say we have a coordinate x with distribution $\mathcal{N}(\mu_{x},\sigma_{x})$, the expected value of a frequency component $sin(2^{k}x),k\in\{0,...,L-1\}$, in PE $\gamma(x)$ can be formulated as:

which can be easily promoted to each frequency component of each coordinate. In short, the expected value of $\gamma(\textbf{x})$ inside frustum $\textbf{v}_{i}$ can be easily calculated in closed form, denoted as:

Then we use the MLP to predict the radiance and density of frustum $\textbf{v}_{i}$, with $\bar{\gamma}(\textbf{v}_{i})$ and $\gamma(\textbf{r})$ as inputs:

By volume rendering the predicted colors and densities of all frustums using Eq.3, we can calculate the radiance returned by a composite cone and use it to represent the corresponding pixel color. Fig.3 illustrates the rendering process of a pixel. The above formulation can be extended to every pixel on the sensor plane and finally approximate the optical effects of thin lens imaging.

So far, we have built concise formulations to implement thin lens imaging in NeRF, but here comes another question: how can we make sure that the MLP can correctly predict the color and density for various sizes of frustum?

A direct solution is to collect enough images taken by a DSLR camera with a zoom lens and apply supervised training [35]. However, this solution is complex and demands extra datasets. Instead, we generate training datasets based on simple blur operations on the original dataset and apply a probabilistic training strategy based on the multiple blurred datasets. We name our training framework the p-training.

We consider a special case that the aperture is set to zero. Since $d_{e}=0$, $d$ is equal to $d_{c}$, i.e., the composite cone is degrade to a cone. Following Eq.7, $d_{c}$ is proportional to $d_{0}$. If we scale $d_{0}$ using scalar $k$, we are essentially using the scaled composite cone to calculate the expected radiance as the pixel’s color. Meanwhile, suppose we use the same scaled Gaussian to calculate the expectation of the pixel’s color on the GT image. In that case, this color should be equal to the rendering color of the composite cone if the MLP can predict correctly by our definition in Eq.12 and Eq.9. Thus we obtain a simple equivalence between rendered results and images. Further, if we introduce different scales of $k$ during training, different scales of composite cones will be involved, which constitutes a simple solution to supervise the MLP for correct prediction.

Specifically, given an original dataset $\textbf{D}=\{\textbf{y}_{1},...,\textbf{y}_{u}\}$, where y denotes the images inside the dataset, we use different sizes of Gaussian blur kernel $\textbf{k}_{j}$, $j\in\{1,...,m\}$ to apply convolution operation (denoted as $\ast$) on every image in dataset D:

which forms a set of multi-blur datasets $\{\textbf{D}_{1},...,\textbf{D}_{m}\}$.

As is shown in Fig.4, $\textbf{D}_{j}\in\{\textbf{D}_{1},...,\textbf{D}_{m}\}$ is randomly chosen with predefined probabilities for each training setp (usually, give datasets with smaller $\textbf{k}_{j}$ the higher chances to be chosen). We use the size of $\textbf{k}_{j}$ to scale the composite cone’s diameter. The rendering process is the same as is described before. Following the practice in mip-NeRF[2], we use a single MLP to implement a two-step coarse-to-fine training. For each composite cone, we first apply a uniform sampling $\textbf{t}^{c}$ to divide frustums and calculate the “coarse” rendered color $\textbf{C}(\textbf{r};\Theta,\textbf{t}^{c})$ and the importance distribution along the composite cone’s axis, Then apply the importance sampling $\textbf{t}^{f}$ to divide frustums and recalculate the “fine” rendered color $\textbf{C}(\textbf{r};\Theta,\textbf{t}^{f})$. The training objective is to minimize the disparity between the blurred GT $\bar{\textbf{C}}(\textbf{r})$ and the composite cone’s rendered color:

where $\Theta$ denotes parameters of the MLP. $\mathcal{B}$ denotes the ray batch. $\bar{\textbf{C}}(\textbf{r})$ is the pixel color in the dataset $\textbf{D}_{j}$, i.e., the blurred pixel color using Gaussian kernel $\textbf{k}_{j}$. $\textbf{t}^{c}$ and $\textbf{t}^{f}$ consist of 128 sampling points, respectively. $\lambda$ is a hyperparameter.

Although our p-training only applied on special cases (aperture diameter as zero and scaled $d_{0}$), experiments show that the trained MLP can generalize to various lens parameters. We guess that this is largely related to the characteristics of IPE. Let’s take Eq.10 for example, where $sin(2^{k}\mu_{x})$ can be seen as the original PE. The essence of IPE lies on the coefficient $\exp(-(2^{k}\sigma_{x})^{2}/2)$, which regulates the IPE’s value range. If x has a more scattered distribution, i.e., high $\sigma_{x}$, the coefficient $\exp(-(2^{k}\sigma_{x})^{2}/2)$ will approach zero. If the distribution is fixed, a component with a higher frequency $k$ will have a smaller value range. Therefore, $\text{E}_{\textbf{x}\sim\mathcal{N}(\boldsymbol{\mu}_{i},\boldsymbol{\Sigma}_{i})}[\cdot]$ can be seen as a low pass filter for $\gamma(\textbf{x})$, considering $\gamma(\textbf{x})$’s different frequency components. Meanwhile, applying $\text{E}_{\textbf{x}\sim\mathcal{N}(\boldsymbol{\mu}_{i},\boldsymbol{\Sigma}_{i})}[\cdot]$ on an image or on scene space will smooth the colors or $\left[\sigma_{i},\textbf{c}_{i}\right]$ pairs. Hence, the training for MLP to satisfy constraint $\text{MLP}(\text{E}[\gamma(\textbf{x})],\gamma(\textbf{r}))=\text{E}[\text{MLP}(\gamma(\textbf{x}),\gamma(\textbf{r}))]$ essentially lets the MLP learn such mapping: let the low-frequency components in IPE map to the blurred color, and gradually approximate the original color as the high-frequency components emerge. This type of mapping is easy for MLP to learn, as is proved in Fig.8.

Consequently, we can use the trained MLP to correctly predict colors and densities given frustums with different sizes and positions as inputs. By which we can render variant defocus effects. For instance, change the focus distance $l$ to move the DOF forward or backward, change the aperture diameter $A$ to raise or reduce the level of defocus effects.

Fig.8 visualizes the training process on the horns dataset (images are resized to 1008$\times$756). Table 1 shows quantitative comparisons.

Though our approach can render various defocus effects, it incurs no additional costs on computation or parameters. Moreover, by simply setting the aperture $A$ as zero, NeRFocus can render large DOF images as normal NeRF can do, with surprisingly close PSNR.

Here we demonstrate various defocus effects that NeRFocus can render at test time. As is well known in photography, increasing the aperture diameter will narrow the DOF (i.e., enhance the defocus effects), while increasing the focus distance will move the DOF forward and vice versa. To verify that NeRFocus’s rendering also conforms to these optical properties, we fix the camera pose and control the aperture diameter $A$ and focus distance $l$ respectively, to better compare the effects caused by $A$ or $l$. Fig.5 and Fig.6 demonstrate the effects that varying focus distance $l$ brings. Fig.7 shows the effects of changing aperture size $A$. We can see that NeRFocus is in good accordance with the mentioned optical properties, which indicates the effectiveness of our optical modeling. Besides, the blur varying along depth shows good continuity, which implies the excellent generalization performance of the MLP trained by our p-training strategy, especially considering that our p-training only uses five different blur kernels.