CG-NeRF: Conditional Generative Neural Radiance Fields

We propose a novel method called conditional generative NeRF (CG-NeRF), which can generate camera-pose-dependent images conditioned on various types of input data. Unlike recent unconditional generative models that learn neural radiance fields from unlabeled 2D images, we extend the generative model to a conditional model utilizing extra information as input, such as text, sketches, gray-scale, low-resolution images, or even color images. We design a model that can generate diverse images with different details, sharing the common characteristics of condition inputs. As shown in Fig. 1, the global feature vector $\bf{c}$ extracted from the input condition is fed to the network along with the noise codes $\mathbf{z}^{s}$ and $\mathbf{z}^{a}$ randomly sampled from a standard Gaussian distribution $p_{z}$. The noise codes specify fine details that are not contained in the given global features. In the proposed model, the generator $G_{\theta}$ (2 in Fig. 1) learns radiance field representations and synthesizes images $\hat{\mathbf{I}}$ corresponding to the given global feature vector $\mathbf{c}$ and noise codes $\mathbf{z}^{s}$ and $\mathbf{z}^{a}$, i.e.,

$$G_{\theta}(\boldsymbol{\xi},\mathbf{c},\mathbf{z}^{s},\mathbf{z}^{a})=\hat{\mathbf{I}},$$

(1)

where $\boldsymbol{\xi}$ is the camera pose for calculating the 3D coordinate $\mathbf{x}$ and the viewing direction $\mathbf{d}$ (Schwarz et al. 2020). Below, we describe the model structure designed for CG-NeRF in detail.

As illustrated in Figure 1, the main architecture consists of three components: a (1) feature extractor $E$ that extracts global feature vectors from the given conditions, (2) a generator that creates an image by reflecting the conditions, and (3) a discriminator that distinguishes real images from fake images based on the condition input and that predicts the camera poses of fake images for the PD loss, which will be described in detail later.

As CG-NeRF aims to synthesize conditional 3D-aware images, the condition input is encoded to a global feature vector through the global feature extractor $E$ (1 in Fig. 1). To extract global semantic features from the given condition inputs in our case, we adopt CLIP (Radford et al. 2021b), accomodating various types of input conditions such as images and text, as a state-of-the-art multimodal encoder.

We design our generator network by combining two recent promising techniques, which are proven to generate high-quality images for the generative neural radiance field task: a SIREN-based backbone  (Chan et al. 2021) and a feature-level volume rendering method  (Niemeyer and Geiger 2021c). The SIREN-based  (Sitzmann et al. 2020) network architecture enhances the visual quality of the NeRF-based generative model but requires a large amount of memory for training due to color-level volume rendering at the full image resolution  (Chan et al. 2021). To address this issue, we leverage feature-level volume rendering, inspired by a recently proposed method  (Niemeyer and Geiger 2021c). The feature-level volume rendering process substantially mitigates the problem because a volume is rendered at the level of feature vector $\mathbf{f}$, having a smaller scale than the image resolution.

Given a global feature vector $\mathbf{c}$, a noise code of shape $\mathbf{z}^{s}$ and appearance $\mathbf{z}^{a}$, the feature fields generator $g_{\theta}$ (2-1 in Fig. 1) produces the density $\sigma$ and feature vector $\mathbf{f}$ in the corresponding $\mathbf{x}$ and $\mathbf{d}$ as

$$g_{\theta}({\bf x}_{pj},{\bf d}_{p},\mathbf{c},\mathbf{z}^{s},\mathbf{z}^{a})=(\sigma_{pj},{\bf f}_{pj}),$$

(2)

where $\sigma_{pj}$ and ${\bf f}_{pj}$ denote the density and the feature vector, respectively, at the corresponding 3D coordinate. Further details are described in the next section.

Once the density $\sigma$ and the feature vector $\bf{f}$ are estimated by the feature fields generator $g_{\theta}$ (2-1 in  Fig. 1) at each 3D coordinate, the final feature $\bf{F}_{p}\in\mathbb{R}^{L_{f}}$ is computed through a feature-level volume rendering process as

$$\mathbf{F}_{p}=\sum_{j=1}^{J}T_{pj}\alpha_{pj}\mathbf{f}_{pj},\\$$

(3)

where the transmittance $T_{pj}=\prod_{k=1}^{j-1}\left(1-\alpha_{pk}\right)$. The alpha value for $\mathbf{x}_{pj}$ is calculated as $\alpha_{pj}=1-e^{-\sigma_{pj}\delta_{pj}}$, and $\delta_{pj}$ is the distance between neighboring sample points along the ray direction (Mildenhall et al. 2020). The 2D feature map $\mathbf{F}\in\mathbb{R}^{H_{V}\times W_{V}\times L_{f}}$ rendered through the volume rendering process is then upsampled to a RGB images at a higher resolution $\hat{\mathbf{I}}\in\mathbb{R}^{H\times W\times 3}$ using the 2D convolutional neural network (CNN) decoder network (2-2 in  Fig. 1). The decoder network consists of CNN layers with leaky ReLU activation functions  (Xu et al. 2015) and nearest neighbor upsampling layers.

We propose a novel approach that aims to disentangle both the shape and appearance contained in a given global feature vector. For a text condition example “ round bird with a red body”, “round” and “bird” are shapes, and “red” is an attribute indicating the appearance. Two mapping networks $M^{s}$ and $M^{a}$ serve to generate the styles of the shape and appearance, respectively, from the global feature vector $\mathbf{c}$ and noise codes $\mathbf{z}^{s}$ and $\mathbf{z}^{a}$. The global feature vector $\mathbf{c}\in\mathbb{R}^{L_{c}}$ contains the prominent attribute of the condition. In constrast, the noise codes $\mathbf{z}^{s}\in\mathbb{R}^{L_{s}}$ and $\mathbf{z}^{a}\in\mathbb{R}^{L_{a}}$ are responsible for the details that the global feature vector does not include. The mapping network consists of pairs of a linear layer and ReLU and produces frequencies $\boldsymbol{\gamma}$ and phase shifts $\boldsymbol{\beta}$ as

		$\displaystyle M^{s}\left(\mathbf{c},\mathbf{z^{s}}\right)=\text{cat}\{\left(\boldsymbol{\gamma}_{i}^{s},\boldsymbol{\beta}_{i}^{s}\right)\}_{i=1\cdots N^{s}}$		(4)
		$\displaystyle M^{a}\left(\mathbf{c},\mathbf{z^{a}}\right)=\text{cat}\{\left(\boldsymbol{\gamma}_{i}^{a},\boldsymbol{\beta}_{i}^{a}\right)\}_{i=1\ldots N^{a}+1}\text{, }$

where $N^{s}$ and $N^{a}$ denote the numbers of MLPs in each block. cat indicates channel-wise concatenation. The predicted frequencies and phase shifts are fed to the two blocks $\Phi^{s}$ and $\Phi^{a}$ in the feature fields generator.

Taking these as inputs along with the 3D coordinate $\mathbf{x}$ and the direction $\mathbf{d}$, two consecutive blocks encode features using pairs of a linear layer and activation function of feature-wise linear modulation (FiLM) SIREN. The sine function of the FiLM SIREN layer modulated by the obtained frequency and phase shift are applied to the outputs of the linear layers as an activation function; i.e.,

$\displaystyle\phi_{i}\left(\mathbf{y}_{i}\right)=\sin(\boldsymbol{\gamma}_{i}(\mathbf{W}_{i}\mathbf{y}_{i}+\mathbf{b}_{i})+\boldsymbol{\beta}_{i})\text{,}$

(5)

where $\phi_{i}$ : $\mathbb{R}^{M_{i}}\mapsto{\mathbb{R}^{N_{i}}}$ is the $i$-th MLP of each ${\Phi}^{s}$ and ${\Phi}^{a}$. $\mathbf{W}_{i}\in\mathbb{R}^{N_{i}\times{M_{i}}}$ and $\mathbf{b}_{i}\in\mathbb{R}^{N_{i}}$ are the weight and the bias applied to input $\mathbf{y}_{i}\in\mathbb{R}^{M_{i}}$. The two blocks in the feature fields generator have the following formulations:

		$\displaystyle\leavevmode\resizebox{258.36667pt}{}{${\Phi}^{s}\left(\mathbf{x}_{pj}\right)=\phi_{N^{s}}^{s}\left(\phi_{N^{s}-1}^{s}(\cdots\phi_{1}^{s}(\mathbf{x}_{pj}))\right)\text{,}$}$		(6)
		$\displaystyle\leavevmode\resizebox{375.80542pt}{}{${\Phi}^{a}({\Phi}^{s}(\mathbf{x}_{pj}),\mathbf{d}_{p})=\phi_{N^{a}+1}^{a}(\text{cat}(\phi_{N^{a}}^{a}(\cdots\phi_{1}^{a}({\Phi}^{s}\left(\mathbf{x}_{pj}\right))),\mathbf{d}_{p}))\text{.}$}$

Inspired by an existing approach (Schwarz et al. 2020), we assign the roles of reflecting the shape to the first block, close to the input, and the appearance to the second block, close to the output. The block for shape utilizes the 3D coordinate as the input to generate shape-encoded features, while the appearance block takes the output of the previous block as input and generates encoded features of the shape and appearance. By utilizing these features and viewing directions as inputs, features reflecting the viewing direction are generated from the last layer of the appearance block.

As our method generates images conditioned on extra inputs, variations of the output images are restricted, especially when a color image is given as a condition input. To enable the generator network to produce semantically diverse images based on the condition input, we regularize the generator network with the diversity-sensitive loss  (Yang et al. 2019a). This is defined as

$\displaystyle\mathcal{L}_{\text{div}}(\theta)=\mathbb{E}_{\mathbf{z}^{s},\mathbf{z}^{a}\sim p_{z},\boldsymbol{\xi}\sim p_{\xi},\mathbf{c}\sim p_{r}}[\parallel\hat{\mathbf{I}}_{1}-\hat{\mathbf{I}}_{2}\parallel_{1}],$

(7)

where $\hat{\mathbf{I}}_{1}$ is $G_{\theta}(\boldsymbol{\xi},\mathbf{c},\mathbf{z}^{s1},\mathbf{z}^{a1})$ and $\hat{\mathbf{I}}_{2}$ is $G_{\theta}(\boldsymbol{\xi},\mathbf{c},\mathbf{z}^{s2},\mathbf{z}^{a2})$.

However, we empirically discover that simply applying the diversity-sensitive loss causes undesirable effects that attempt to change not only the style but also the pose of the output images (Fig. 5). Because the pose of the output images should be determined only by the input camera pose $\boldsymbol{\xi}$, pose changes in the output images are a significant side effect. We analyze this undesirable phenomenon as follows; from the generator network’s point of view, the model maximizes the pixel difference via two different methods: (1) changing the style of the output images as desired or (2) changing the poses between two output images generated with the same camera pose, which is strongly undesired.

To explicitly address such an issue, we propose a pose regularization term applicable to the original diversity-sensitive loss, which explicitly penalizes pose difference between images generated from different noise codes $\mathbf{z}^{s}$ and $\mathbf{z}^{a}$ but from the same camera pose. The intuition behind the proposed regularization is that the model generates two images to have only a style difference constrained to have the same pose, which can be additionally learned by an auxiliary network. We propose to add the regularization term $\mathcal{L}_{\text{pose}}$ to the diversity-sensitive loss $\mathcal{L}_{\text{div}}$, which is defined as

$\mathcal{L}_{\text{pose}}(\theta)=\mathbb{E}_{\mathbf{z}^{s},\mathbf{z}^{a}\sim p_{z},\boldsymbol{\xi}\sim p_{\xi},\mathbf{c}\sim p_{r}}[1-\cos(D_{\psi}^{\xi}(\hat{\mathbf{I}}_{1})-D_{\psi}^{\xi}(\hat{\mathbf{I}}_{2}))],$

(8)

where $D_{\psi}^{\xi}$ is the auxiliary pose estimator network we additionally train for the pose penalty loss jointly with the discriminator.

The proposed method simultaneously learns the output images’ poses by training the pose estimator network. We modify our discriminator network to contain an auxiliary pose estimator, by adjusting the channel size of the last layer to estimate the camera pose values of the output image. Because we randomly sample camera poses $\boldsymbol{\xi}$ from the prior distribution $p_{\xi}$ to generate view-consistent images, the sampled camera pose is utilized as the ground truth pose when training the pose estimator. We define the camera pose $\boldsymbol{\xi}$ with radius $r_{cam}$, rotation angle $\kappa_{r}\in[-\pi,\pi]$, and elevation angle $\kappa_{e}\in[0,\pi]$. Given that we use a fixed value for $r_{cam}$=1, the pose estimator predicts the rotation angle and elevation angle, applying the Sigmoid function to the output value multiplied by $2\pi$ and $\pi$ respectively. The camera pose reconstruction loss is defined as

$\displaystyle\leavevmode\resizebox{375.80542pt}{}{$\mathcal{L}_{\text{pose}}(\psi)=\mathbb{E}_{\mathbf{z}^{s},\mathbf{z}^{a}\sim p_{z},\boldsymbol{\xi}\sim p_{\xi},\mathbf{c}\sim p_{r}}[1-\cos(D_{\psi}^{\xi}(\hat{\mathbf{I}})-\boldsymbol{\xi}_{\text{gt}})],$}$

(9)

where $D_{\psi}^{\xi}(\hat{\mathbf{I}})=\boldsymbol{\xi}_{\text{pred}}=(\hat{\kappa_{r}},\hat{\kappa_{e}})$. $D_{\psi}^{\xi}$ denotes the auxiliary pose estimator and $\boldsymbol{\xi}_{\text{gt}}$ is a randomly sampled camera pose value to generate $\hat{\mathbf{I}}$. Because the angle can be represented by a periodic function, we design the pose reconstruction loss with the cosine function to penalize the angle difference, addressing its discontinuity at $2\pi$.

To synthesize conditional outputs, we adopt a conditional GAN (Isola et al. 2017) by training a discriminator that learns to match images and condition feature vectors. As shown in Figure 1, the discriminator extracts the image feature through a series of 2D convolution layers, and the image feature is then concatenated with matching condition $\mathbf{e}$ to predict the condition-image semantic consistency. The matching condition $\mathbf{e}\in\mathbb{R}^{L_{c}+L_{s}+L_{a}}$ is the global feature vector $\mathbf{c}$ concatenated with detail codes $\mathbf{z^{s}}$ and $\mathbf{z^{a}}$. The number of feature extracting layers is determined by the resolution of the training images. The discriminator network learns whether the given image is real or fake and matches its condition feature vector simultaneously.

At training time, we use the non-saturating GAN loss with a matching-aware gradient penalty (Mescheder, Geiger, and Nowozin 2018; Tao et al. 2020). Instead of the $R_{1}$ gradient penalty (Mescheder, Geiger, and Nowozin 2018), we adopt the matching-aware gradient penalty loss, which is known to promote the generator to synthesize more realistic and semantic-consistent images to condition-image pairs. We define three different types of data items: synthetic images with the matching condition, real images with a matching condition, and real images with a mismatching condition. The target data point on which the gradient penalty is applied can be defined by real images with the matching condition feature vector. The entire formulation of conditional GAN loss, i.e.,

		$\displaystyle\mathcal{L}_{\text{adv}}(\psi)=\mathbb{E}_{\mathbf{I}\sim p_{r}}\left[f\left(D_{\psi}(\mathbf{I},\mathbf{e})\right)\right]$		(10)
		$\displaystyle+(1/2)\mathbb{E}_{\mathbf{I}\sim p_{mis}}\left[f\left(-D_{\psi}(\mathbf{I},\mathbf{e})\right)\right]$
		$\displaystyle+(1/2)\mathbb{E}_{\boldsymbol{\xi}\sim p_{\xi},\mathbf{e}\sim p_{r},p_{z}}\left[f\left(-D_{\psi}\left(G_{\theta}(\boldsymbol{\xi},\mathbf{c},\mathbf{z}^{s},\mathbf{z}^{a}),\mathbf{e}\right)\right)\right]$
		$\displaystyle+k\mathbb{E}_{\mathbf{I}\sim p_{r}}\left[\left(\left\|\nabla_{\mathbf{I}}D_{\psi}(\mathbf{I},\mathbf{e})\right\|+\left\|\nabla_{\mathbf{e}}D_{\psi}(\mathbf{I},\mathbf{e})\right\|\right)^{p}\right],$
		$\displaystyle\mathcal{L}_{\text{adv}}(\theta)=\mathbb{E}_{\boldsymbol{\xi}\sim p_{\xi},\mathbf{e}\sim p_{r},p_{z}}\left[f\left(D_{\psi}\left(G_{\theta}(\boldsymbol{\xi},\mathbf{c},\mathbf{z}^{s},\mathbf{z}^{a}),\mathbf{e}\right)\right)\right]$

where $f(u)=-\log(1+\exp(-u))$. $p_{r}$ and $p_{mis}$ denote the real data distribution and mismatching data distribution, respectively. $k$ and $p$ are two hyper-parameters that balance the gradient penalty effects.

Our full training objective functions for the generator network $G_{\theta}$ are summarized as

$\displaystyle\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{adv}}-\lambda_{\text{div}}\mathcal{L}_{\text{div}}+\lambda_{\text{pose}}\mathcal{L}_{\text{pose}},$

(11)

where $\lambda_{\text{div}}$ and $\lambda_{\text{pose}}$ are weights for each loss term.