Learning Unified Decompositional and Compositional NeRF for
Editable Novel View Synthesis

Two-branch network architecture design. For scene decomposition in the fine-stage, we design a two-branch network architecture with a shared MLP backbone to learn generic implicit scene representation from input point coordinates (see Fig. 3). Specifically, the shared backbone takes a sampled 3D point coordinate $\mathbf{p}=(x,y,z)$ and view direction $\mathbf{d}$ of a camera ray as input, and predicts a generic scene feature vector $\mathbf{f}$. Similar to [37], we also apply a set of learnable codes, each representing the background or an object. We further pass $\mathbf{f}$ accompanying with background or object codes to the following MLP heads. For better background learning, we use separate the background head and the object head, predicting an RGB color $\mathbf{c}_{\text{object}}^{i}$ and a volume density $\sigma_{\text{object}}^{i}$ for the corresponding background or object $i$. For simplicity, we refer to the background as a particular object with index $i=1$. For efficient point sampling, we apply a coarse-to-fine sampling technique. In the coarse stage, we only adopt several MLPs after the backbone for predicting RGB color $\mathbf{c}_{\text{guidance}}$ and volume density $\sigma_{\text{guidance}}$ of the global scene. We use the distribution of coarse prediction for the fine point coordinate sampling, so we call these MLPs guidance head. In the fine stage, we also use the prediction of the guidance head as a residual radiance field in the following composition module.

3D one-hot density regularization. To achieve effective decomposition of objects/background radiance fields from the scene, we also design a one-hot regularization term for the uncertain regions in the background radiance field. Our motivation is that the 3D points belonging to the foreground objects should not have any volume density in the background radiance field. We thus apply a one-hot regularization loss term, which acts as an additional constraint to the object-specific radiance field optimization objective $\mathcal{L}_{\text{object}}$. In detail, we first obtain the 3D point positions that foreground objects occupy by determining whether the foreground objects take the largest volume density at the points. We thus define a binary occupancy mask $M_{c}$, and apply a regularization term for the occupied points as follows:

2D in-painting pseudo supervision. To model a better background-specific radiance field, we need to learn what the scene looks like even if all the foreground objects are removed from the scene. Several occluded regions in the background can be seen in other views, while regions covered by the objects are invisible from all views, which are uncertain regions. We thus do not have any pixel-wise supervision for those regions in the training data, resulting in unsatisfactory extrapolation results on those regions when we learn the background radiance field. To tackle this problem, we involve a pre-trained in-painting model to fill those uncertain regions on the 2D images. Our motivation is that the 2D in-painting model can utilize local context information on an image to fill the regions, providing effective 2D pseudo color supervision. Specifically, we use a famous pre-trained lama  [31] model to fill the uncertain 2D background regions and then apply a color loss for the rendered background pixels with a foreground mask $M_{1}(\boldsymbol{r})^{1}$:

$\displaystyle\mathcal{L}_{\text{color\_pseudo}}=M_{1}(r)^{1}\|\hat{C}(\boldsymbol{r})_{\text{object}}^{1}-C(\boldsymbol{r})_{\text{pseudo}}\|_{2}^{2}$

(2)

Intermediate local object rendering. To enforce the decomposition of all object-specific radiance fields, we apply a color consistent loss and an opacity loss with semantic masks as  [18, 37] via rendering both 2D color and opacity:

	$\displaystyle\hat{C}(\boldsymbol{r})_{\text{object}}^{k}$	$\displaystyle=\sum_{i=1}^{N}T_{i}^{k}\big{(}1-\exp(-\sigma_{\text{object}}^{k}\delta_{i})\big{)}\mathbf{c}_{\text{object}}^{k},$		(3)
	$\displaystyle\hat{O}(\boldsymbol{r})_{\text{object}}^{k}$	$\displaystyle=\sum_{i=1}^{N}T_{i}^{k}\big{(}1-\exp(-\sigma_{\text{object}}^{k}\delta_{i})\big{)}$

where $T_{i}^{k}=\exp(-\sum_{j=1}^{i-1}\sigma_{\text{object}}^{k}\delta_{j})$ is the transmittance representing how much light is transmitted along the ray for the $k$-th object; $N$ is the number of sampled points along the camera ray, and $\delta_{i}$ is the distance between two adjacent points. Then we apply the following losses:

	$\displaystyle\mathcal{L}_{\text{object}}=$	$\displaystyle\sum_{k=1}^{K+1}M_{o}(\boldsymbol{r})^{k}\|\hat{C}(\boldsymbol{r})_{\text{object}}^{k}-C(\boldsymbol{r})\|_{2}^{2}$		(4)
		$\displaystyle+\lambda M_{b}(\boldsymbol{r})^{1}\|\hat{O}(\boldsymbol{r})_{\text{object}}^{1}-M_{o}(\boldsymbol{r})^{1}\|_{2}^{2},$

where $M_{o}(\boldsymbol{r})$ and $M_{b}(\boldsymbol{r})$ are 2D spatial weight maps determined by object-level and background-level semantic masks, respectively, which indicates that rendered object-specific pixels are given supervision at the corresponding semantic mask region. The background region in the binary semantic mask can be either real background or occluded by a foreground object, so we use a small weight value (i.e. 0.05) in $M_{b}$ as a balance weight for the uncertain region.

Gumbel-Softmax one-hot object radiance activation. As shown in  Fig. 3, for each sampled point, there are $K+1$ predictions in the local radiance field decomposition. Considering that each point in the 3D space can only belong to either one object or the background, we use the predicted $(\mathbf{c}_{\text{object}}^{i},\sigma_{\text{object}}^{i})$ with the maximum volume density $\sigma_{\text{object}}^{i}$ among all the object instances to construct an activated object-specific radiance field. More specifically, to address the gradient backpropagation problem of the argmax operation, following  [10, 16], We employ the Gumbel-Softmax function on volume density as a continuous and differentiable approximation. First, we calculate a soft probability $p_{i}$ for the $i$-th object branch as

$\displaystyle p_{i}=\frac{\exp^{((\sigma_{\text{object}}^{i}+g_{i})/tau)}}{\sum_{j}\exp^{((\sigma_{\text{object}}^{j}+g_{j})/tau)}}$

(5)

The above $g_{1},g_{2},..,g_{j}$ are the independent and identically distributed samples drawn from $\operatorname{Gumbel}(0,1)^{1}$ [10]. The symbol $tau$ is the temperature ratio set to 0.1 in our experiments. Then, we can obtain one-hot composition weights from $p_{i}$ for combining the predictions of all the object/background branches, which ensures one-hot activation for all the object/background radiance fields, and enables effective backpropagation for learning object-specific radiance fields and their composition. Specifically, we calculate the one-hot weights $\mathbf{M}_{{obj}}$ as Eq. 6. Let us denote $\mathbf{P}$ as the soft probability vector, containing $K+1$ predictions calculated by Eq. 5. $\mathbf{M}_{{obj}}$ is a one-hot vector of the $K+1$ elements. The one-hot vector is calculated with a stop gradient operation $f_{sp}$ on $\mathbf{P}$, which not only reserves the one-hot activation but also allows the gradient backpropagated through $\mathbf{P}$. Then, the computation of $\mathbf{M}_{obj}$ is written as:

Residual radiance field composition. After the one-hot object radiance activation, we further combine the composited radiance fields from objects/background branches with the predicted residual radiance field, to produce a final composited scene radiance field, using point-wise average pooling operation applied on the color and density. Then we apply a color loss $\mathcal{L}_{\text{comp}}$ for the pixels rendered from the composited radiance field.

Overall framework optimization. The overall optimization loss for training the joint framework consists of the following four different loss functions:

$\displaystyle\mathcal{L}_{\text{overall}}=\mathcal{L}_{\text{comp}}+\lambda_{1}\mathcal{L}_{\text{object}}+\lambda_{2}\mathcal{L}_{\text{color\_pseudo}}+\lambda_{3}\mathcal{L}_{\text{one-hot}},$

(8)

where, $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are the weights for balancing the learning with the different loss functions.

Novel View Rendering and Scene Editing After training the joint framework, we can render images of arbitrary perspectives based on the composited radiance field. By re-organizing the object radiance fields, we can generate images with object manipulations, including removal, addition, duplication, and changing position (e.g., rotation).

Datasets. We use ToyDesk  [37] and ScanNet  [6] datasets for the evaluation. ToyDesk consists of two sets of indoor multi-view images. In each scene, several toys are put on a desk in the center of the room. The multi-view images are captured by looking at the desk center with 360 degrees around the desk, and the camera poses and meshes are provided. The instance semantic labels are annotated on object meshes and then projected onto each perspective to obtain their corresponding 2D semantic label maps. ScanNet is an RGB-D video dataset that contains more than 1500 scans of indoor scenes for multiple 3D understanding tasks. Handheld RGB-D scanners are used for scanning various indoor scenes. Rich annotations such as camera pose, semantic segmentation, and surface reconstruction are also provided. Following ObjectNeRF [37], we choose four scenes, i.e., 0024_00, 0038_00, 0113_00, and 0192_00, from the whole dataset for the experiment.

Implementation details. Our joint framework consists of a shared backbone, background head, object head, and guidance head. The share backbone is a 4-layer Multi-Layer Perceptron (MLP), and the output generic implicit scene feature has a dimension of 256. The follow-up heads are all three-layer MLPs with a color and density prediction layer to generate corresponding radiance fields. Note that our one-hot regularization and activation module has no parameters to be optimized. We also utilize positional encoding to increase the representation ability for high-frequency details. Specifically, we adopt ten levels for coordinates and four levels for ray directions. The framework is trained end-to-end from scratch, without voxel feature initialization as in [37] or geometric initialization as in [35].

Baseline Models. To verify the effectiveness of our designs for scene decomposition and composition, We conduct several ablation studies considering different variants: (i) ’w/o 3D One-hot Density Regularization’ means that we remove the one-hot regularization term for the uncertain regions in the background radiance field. (ii) ’w/o Gumbel-Softmax Activation’ indicates that we do not use the Gumbel-Softmax one-hot weight for compositing the object radiance fields. (iii) ’w/o In-painting Supervision’ means that we do not apply the in-painting pseudo supervision for the background radiance field. (iv) ’w/o Residual Radiance Composition’ means that we do not use the guidance radiance field for fine sampling and remove the residual radiance composition in the fine stage.

Effect of 3D one-hot density regularization. The quantitative results based on different evaluation metrics, i.e., PSNR, SSIM, and LPIPS, are shown in Table 1. Several examples of rendered images from the learned background radiance field are presented in Fig. 5. We evaluate the effectiveness of 3D one-hot density regularization by disabling the loss term. As can be observed from Table 1, our joint full mode with 3D one-hot density regularization achieves better results than the model without it.

Effect of Gumbel-Softmax object radiance activation. We also investigate the effect of using Gumbel-Softmax one-hot activation for the composition of background/object radiance fields. We use an alternative composition method, in which all the local object/background radiance fields that contribute to the same 3D points in the composited field, are directly added instead of using the Gumbel-Softmax one-hot activation weights, as introduced in the method part. As shown in Fig. 5c, without the Gumbel-Softmax one-hot activation, the model cannot successfully learn a decomposed background radiance field (all black), confirming our motivation of using the Gumbel-Softmax activation for a more effective scene representation in a unified framework.

Effect of 2D in-painting pseudo supervision and residual radiance composition. We also show the contribution of the proposed 2D in-painting pseudo-supervision. If we directly disable the pseudo supervision in the framework, the qualitative results are slightly lower, as shown in the next to last row in Table. 1. At the same time, as can be seen in Fig. 5, lacking pseudo supervision suffers from significantly downgraded extrapolation in the region occupied by the foreground objects in the 3D space, which indicates that the proposed scheme can effectively help the local object/background disentanglement, especially for learning the radiance field representation of the background on uncertain regions. As shown in Table. 1 and Fig. 5, if we remove the residual radiance composition, the overall qualitative metrics and background decomposition both decline, which demonstrates the importance of using a global radiation field to provide coarse sampling guidance and also further merged as residual information in the fine stage.