Neural Impostor:
Editing Neural Radiance Fields with Explicit Shape Manipulation

When it comes to geometric editing of 3D content, the core of the problem is to encode and manipulate the spatial information of the scene. The spatial encoding process of NeRF is to map the low-dimensional smooth inputs such as position and direction to a high-dimensional feature space that can be trained to encode high-frequency details. Initially, researchers use a fully connected MLP with positional encoding [MST^∗20] to encode the spatial information of the entire scene, which synthesizes realistic novel views with fine details, but also suffers from high computational cost when querying the large neural network for every sample. To enhance the training efficiency, hybrid spatial representations utilizes the sparsity of spatial data, factorize the scene into explicitly stored features and implicit neural functions. Depending on the underlying structure, we categorize spatial encoding methods into grid-based and mesh-based structures. Grid-based encoding schemes, here we refer to pre-constructed regular spatial structures, such as sparse voxel grids [LGL^∗20, HSM^∗21], octree grids [YLT^∗21, YFKT^∗21], triplanar structures [CLC^∗21], codebooks [TET^∗22], tensorial decomposition [CXG^∗22] and multi-grid hash encoding [MESK22]. The grid-based approaches provide efficient querying of features stored in the grid, which greatly improves the training and rendering speed. Mesh-based encoding, on the other hand, uses a mesh-like structure with more flexible topologies, such as point cloud [XXP^∗22](point clouds can be seen as unconnected mesh vertices), triangle soup [CFHT22], manifold mesh [CJH^∗22, WXB^∗23], duplex mesh [WRB^∗23] and tetrahedral mesh [GCX^∗20]. Mesh-based encoding provides additional flexibility in manipulating implicit fields; however, its irregular structure necessitates a robust adaptive mesh generator. Furthermore, when storing all the features on explicit meshes, mesh-based encoding often lacks volumetric appearances.

In this paper, we aim to achieve both the efficiency of grid-based encoding and the flexibility of mesh-based encoding, therefore enables the geometric editing of NeRF while maintaining the high-quality volumetric rendering. In the proposed method, we utilize a tetrahedral based proxy, and bounds the explicit mesh and implicit field with a generalized barycentric encoding method.

Using explicit proxies to manipulate 3D models is a common practice in computer graphics. Proxies usually retain a lower dimensional representation of the original model, which serve as a model reduction process and can be used to accelerate the rendering process or provide a more intuitive interface for users to manipulate the model. For example, in geometric modeling and shape manipulation, proxies reduces the complexity of original models, therefore enables structure preserving shape manipulation [ZFCO^∗11], shape collection synthesis [XZCOC12] and image-based geometric modeling [XZZ^∗11]. Bounding volumes, which are one of the simplest form of proxies, are widely used in collision detection [GLM96, KHM^∗98], ray tracing [MOB^∗21] and shape approximation [CB17]. Proxy-based methods are also used in manipulating NeRF, one of the commonly used proxies is the tetrahedral mesh, which is used to approximate the geometry of the scene. By mapping the samples in the deformed tetrahedral mesh to the canonical space [GKE^∗22, XH22, YSL^∗22, PYL^∗22], one can manipulate the neural implicit field with geometric operations on the explicit tetrahedral mesh. In this paper, we follow the general idea of using tetrahedral mesh as proxy, but instead of bending sampling rays to the canonical space, we propose a generalized barycentric encoding method efficiently store the implicit field in the tetrahedral mesh, which allows more flexible operations, please see Sec. 4. Worth to mention that there is a special type of proxy called impostor[Jes05], which originally is a 2D image that approximates the appearance of a 3D model from a specific viewpoint by interpolating pre-rendered multi-view appearances. The concept of impostor is later extended to 3D models, such as [JW02, YD08], and is widely used in real-time rendering of large-scale scenes. In this paper, we introduce the term, "Neural Impostor," which is derived from the concept of Impostor. Our proposed method entails the creation of an editable 3D proxy that approximates the volumetric appearance of a 3D model from arbitrary viewpoints, just like traditional impostor did in real-time rendering.

Recently, Editing Neural Radiance Fields has seen significant progress in terms of reconstructing and modifying various aspects of appearance, such as relighting, controlling shapes, and altering colors or palettes of objects[LZZ^∗21, KLB^∗22]. Some techniques [YSL^∗22, GKE^∗22, JKK^∗23] have enabled modification of scene parts and their respective locations, thus providing more control over the rendered scene. However, editing the dynamics of moving objects, especially with topological changes, has been a major challenge. These changes can lead to motion discontinuities, causing noticeable artifacts if not properly modeled[YSL^∗22]. [KYK^∗21] has tried to mitigate these challenges using manual supervision, but its capabilities are limited to one-dimensional editing per scene part, necessitating user annotations for supervision. In contrast, EditableNeRF[ZcLX22] uses an image sequence from a single camera to train a network, modeling topologically varying dynamics via surface key points. Users can edit the scene by manipulating these key points, enabling more intuitive multi-dimensional editing (up to 3D). NeuralEditor[CLW23] is another novel approach that leverages the explicit point cloud representation underlying NeRFs for shape editing tasks. It employs a unique rendering scheme based on deterministic integration within K-D tree-guided density-adaptive voxels, which results in high-quality rendering and precise point clouds. Despite these advancements, editing capabilities within a computed NeRF scene remain relatively limited, especially when compared to traditional CGI workflows. Recent progress in pre-trained large-scale models has enabled rapid progress in a brand new instruction or prompt based creation and interaction with digital contents. For instance, Instruct-NeRF2NeRF[HTE^∗23] leverages an image-conditioned diffusion model to iteratively edit input images while simultaneously optimizing the underlying radiance field of the 3D scene, resulting in a realistic 3D scene that adheres to the provided editing instructions. [MPE^∗23] offering genuine, personalized, temporal consistent 3D avatar edit by combining text-to-image diffusion model with neural radiance field (NeRF) through an innovative sampling strategy to incorporates multiple keyframes representing different camera viewpoints and timestamps into a single diffusion model. The edits are made in a canonical space and propagated to remaining time steps through a pretrained deformation network.

In this study, we revisit the issue of geometry editing using a new localized retraining technique that allows for the flawless integration of geometry manipulation tasks from modern 3D modeling software into Neural Radiance Fields. This improves NeRF’s suitability as a representation for User Generated Content (UGC) and AI Generated Content (AIGC).

As shown in Figure Neural Impostor: Editing Neural Radiance Fields with Explicit Shape Manipulation, to construct an editable implicit radiance field, we reintroduce an explicit representation of its geometric form while modeling the radiance field representing the appearance. That is, a coarse tetrahedron mesh that acts as the boundary for shape editing, also known as the "cage" [JSW05, JMD^∗07]. Unlike those traditional cage-based shape editing methods, the tetrahedral cage in our NeRF model does not have a fine mesh embedded in—after all, the bare bones of a NeRF model lie in its implicit representation of the radiance field. Therefore, within each tetrahedron of the cage, we use MLPs to represent its view-dependent radiance field distribution, maintain a separate radiance field in each tetrahedron of the cage, and convert the position encoding under the original Cartesian coordinate system into the barycentric coordinate space defined by the cage. At the same time, because the definition of distance and angle under barycentric coordinates is not clear, we retain the original direction encoding method of NeRF under the Cartesian coordinate system, to model the view-dependent effect by introducing the direction of light in the subsequent decoding process (as seen in Figure 1).

Specifically, we use $M=(V,T)$ to represent the tetrahedron mesh of the Neural Impostor, where $V=(v_{1},v_{2},...,v_{N})$ represents the set of $N$ vertices of the tetrahedron mesh, and $T=(t_{1},t_{2},...,t_{K})$ represents the $K$ tetrahedron that make up the tetrahedron mesh. Each tetrahedron $t_{k}\in T$ is represented by its four vertices $V_{t}={v_{0}^{k},v_{1}^{k},v_{2}^{k},v_{3}^{k}}$. For each position coordinate $p$ in Cartesian space, we map it to the corresponding tetrahedron $t$ and the corresponding barycentric coordinates $\Lambda_{t}={\lambda_{0},\lambda_{1},\lambda_{2},\lambda_{3}}$ using the barycentric coordinate query function $\mathsf{Q}(\cdot)$ to construct the encoding space, that is:

		$\displaystyle\mathcal{G}_{M}=\bigcup_{t\in T}\Lambda_{t}$		(2)
		$\displaystyle\Lambda_{t}=\Bigl{\{}\{\lambda_{0}^{t},\lambda_{1}^{t},\lambda_{2}^{t},\lambda_{3}^{t}\}\textit{ }\mid\textit{ }0\leq\lambda_{i}^{t}\leq 1\textit{ and }\sum_{i}\lambda_{i}^{t}=1\Bigl{\}}$
		$\displaystyle\mathsf{Q}:\mathbb{R}^{3}\to\mathbb{R}^{5},\quad\mathsf{Q}(p)=(t,\Lambda_{t}),\quad\textit{and }p=\Lambda_{t}V_{t}$

during rendering, $p$ can be transformed to be represented in a ray coordinate system with the ray origin $\dot{o}$ as the coordinate origin and the ray direction $\vec{d}$ as the $z$-axis. Considering the good linear interpolation characteristics of barycentric coordinates, for any point $p$ inside the tetrahedron, we can obtain it by linearly interpolating the barycentric coordinates of the incident and exit points of the ray relative to the tetrahedron, namely:

		$\displaystyle\Lambda_{t}^{p}=\alpha\Lambda_{t}^{p_{0}}+(1-\alpha)\Lambda_{t}^{p_{1}},$		(3)
		$\displaystyle\textit{where }p_{0}=\dot{o}+t_{0}\vec{d},\quad p_{1}=\dot{o}+t_{1}\vec{d},\quad\textit{and }t_{0}<t_{1}$

where $\alpha\in[0,1]$ determines the relative distance of $p$ to the entrance/exit points. Finally, we pack the barycentric coordinates of each sampling point $\Lambda_{t}^{p}$, the tetrahedron index $t$, and the light ray direction in the Cartesian coordinate system $\vec{d}$ together to represent a sample in the encoded space $\mathcal{G}$.

Apart from enabling shape editing (see Section 4.1), the tetrahedral cage can be also used as a proxy for collision processing. It allows the use of full-fledged collision processing algorithms to detect and resolve collisions between a NeRF model and other objects—an essential component for bringing NeRF models in a physical simulation. Last but not least, the tetrahedral cage offers a natural data structure that helps to speed up the image rendering process. Typically, the NeRF rendering algorithm samples a set of points along each camera ray and accumulates a color value. With the cage, this process can be implemented more efficiently.

NeRF[MST^∗20] treats the target scene as a volume space with a certain density, and the physical quantities of discrete sampling points are approximated by integration using body rendering. Within a certain limit of the number of samples, in order to accurately model the scene details, we want the sampling points to be concentrated as much as possible in places that contribute to the pixel color to avoid computational waste, i.e., the sampling density should approximate the real density distribution of the scene as much as possible. For this reason, in NeRF[MST^∗20], researchers designed a sampling method based on probability distribution, i.e., first obtain the approximate density distribution in space by coarse-grained uniform sampling, then perform fine-grained sampling according to the probability density of coarse-grained sampling points, and fuse the coarse and fine-grained sampling points for the integration process of body rendering. This process involves querying the spatial density decoder for both coarse- and fine-grained sampling, and therefore greatly reduces the rendering effectiveness.

Instant-NGP[MESK22] accelerates the rendering process with an explicit spatial occupancy grid that roughly marks the empty and non-empty states of space using binary bits. The sampling process then skips over sparse space (i.e., empty space), thereby concentrating the sample points near the surface. This method of sampling based on the spatial occupancy grid can greatly speed up the rendering process in NeRF. However, this approach also requires the spatial occupancy grid to be updated at a certain frequency during the training process, namely re-querying the MLP to obtain the spatial density of the grid vertices. Besides, the misalignment between the voxel grid structure defining the occupancy field and the tetrahedron grid may cause jittering or ghosting artifacts.

Therefore, we designed a sampling method based on barycentric coordinates. According to the truncation theorem[MESK22], the sampling step length during light propagation should be proportional to the propagation distance along the ray. This proportion is determined by the predefined cone angle $\theta$. Considering that the size of each tetrahedron may change significantly during the deformation process, while the barycentric coordinate distances defined by the incident and outgoing points of the tetrahedron remain relatively stable, we use the interpolation weight $\alpha$ from Equation (3) to replace the step length in the Cartesian coordinate system, which means:

$$\Delta\alpha_{i+1}=\Delta\alpha_{i}+\theta\cdot A_{i}$$

(4)

where $A_{i}=\sum_{i}\Delta\alpha_{i}$ represents the cumulative sampling distance starting from the incident point of the first valid tetrahedron. Consequently, the sampling process can be described as:

		$\displaystyle\mathsf{S}:\mathbb{R}^{6}\rightarrow\mathbb{R}^{N\times 8}$		(5)
		$\displaystyle\mathsf{S}(\dot{o},\vec{d})=\Biggl{\{}\{t,\Lambda_{t},\vec{d}\}_{\times N}\ |\ t\in T,\Lambda_{t}\in\mathbb{R}^{4},0\leq\Lambda_{t}\leq 1\textit{ and }\|\Lambda_{t}\|_{1}=1\Biggr{\}}$

where $\dot{o},\vec{d}$ represent the origin and direction of the ray, $N$ stands for the number of sample points for each ray, and $(t,\Lambda_{t})$ respectively represent the index of the tetrahedron to which the sample point belongs and its 4-dimentional barycentric coordinates. After sampling, each sample point is represented as a set composed of tetrahedron index, barycentric coordinates, and ray direction.

In every single tetrahedron, we need to store a view-dependent neural radiance field. Furthermore, the radiance field must transform smoothly when the tetrahedron is deformed to avoid flickering artifacts. With this in mind, we represent the positional quantity of the neural radiance field in barycentric coordinates. As the light ray direction of the neural radiance field is a deformation-independent quantity, we encode the light ray direction in the form of spherical harmonic functions in original Cartesian coordinate system. The barycentric coordinate system is a local linear space relative to each tetrahedron. When we change the tetrahedron’s vertex positions, points with fixed barycentric coordinates will move correspondingly. Therefore, under the encoding method of the Neural Impostors based on the tetrahedron barycentric coordinates, when the tetrahedron acting as a proxy deforms, the neural radiance field encoded by each tetrahedron will also undergo corresponding shape changes. Represent the tetrahedron lookup function as $\mathsf{Q}$, local coordinate encoder inside tetrahedron $t$ as $\mathsf{E}_{p}^{t}$ and the global direction encoder as $\mathsf{E}_{d}$, then the encoding process of the Neural Impostors at sampling point $p$ can be represented as:

		$\displaystyle\mathsf{Q}:\mathbb{R}^{3}\rightarrow\mathbb{I},\quad t=\mathsf{Q}(p)$		(6)
		$\displaystyle\mathsf{E}_{p}^{t}:\mathbb{R}^{4}\rightarrow\mathbb{R}^{F\times L},\quad f_{p}=\mathsf{E}_{p}^{t}(\Lambda_{t}^{p})=h_{t}(\Lambda_{t}^{p})$
		$\displaystyle\mathsf{E}_{d}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{D^{2}},\quad f_{d}=\mathsf{E}_{d}(\vec{d})=\textit{SH}_{D}(\vec{d})$

where $\Lambda_{t}^{p}$ is the barycentric coordinate of the sampling point $p$ inside the tetrahedron $t$, $L$ is the number of levels in the multiresolution hash, $F$ is the number of features in each level, $D$ is the order of the spherical harmonic function encoding, $h_{t}$ represents the feature querying processing from tetrahedron $t$’s local hash table, and $\textit{SH}_{D}$ is the $D$-order spherical harmonic function. After encoding, we can obtain the position feature $f_{p}$ and direction feature $f_{d}$ corresponding to each sampling point $p$.

However, the use of barycentric coordinates complexes the encoding process (i.e., $\mathsf{E}_{p}^{t}$ in Equation (6)). We wish to leverage the multi-resolution hash encoding scheme proposed in  [MESK22], as it offers high rendering quality and low memory footprint. This encoding scheme conceptually stores trainable features on a multi-resolution Cartesian grid, and directly transferring this encoding scheme in barycentric coordinates requires the creation of a multi-resolution tetrahedron mesh within a tetrahedron and the storage of feature vectors on the tetrahedron mesh’s vertices. Yet, this approach is troublesome. When a tetrahedron is subdivided for finer level mesh creation, it results in four smaller tetrahedron and an octahedron in the center. It is unclear how to split the octahedron into a set of tetrahedron in a consistent way. More importantly, when it comes to image rendering, a recurring step is to find, on each multi-resolution level, the tetrahedron in which a sampled point on a camera ray is located. This, however, is computationally much more expensive than its counterpart in [MESK22] (i.e. finding in an axis-aligned Cartesian grid the voxel in which a sampled point is located).

Instead, we propose to maintain trainable features not on any tetrahedral vertices but on a four-dimensional (4D) grid associated with the tetrahedron. The 4D grid is multiresolution, constructed in the following way. Consider a point $p$ inside the tetrahedron $t$ with barycentric coordinates $\Lambda_{t}(p)={\lambda_{0}^{t},\lambda_{1}^{t},\lambda_{2}^{t},\lambda_{3}^{t}}$. Although its barycentric coordinates only have three degrees of freedom (i.e., constrained by $\sum_{i=0}^{3}\lambda_{i}^{t}=1$), algebraically it can be regarded as a 4D point located inside a 4D voxel. This 4D voxel has 16 vertices (corners), each with a coordinate $(u_{1},u_{2},u_{3},u_{4})\in{0,1}^{4}$. This voxel forms the highest-level grid with a resolution of $1\times 1\times 1\times 1$. We create the multiresolution grid by progressively subdividing this highest-level grid, and bind the trainable features characterizing the radiance field of the tetrahedron to the nodes of the multiresolution grid.

The relation of a tetrahedron to its 4D grid can be interpreted geometrically. On the highest-level grid—which is a single 4D voxel—there are 16 voxel vertices. Four of them have coordinates satisfying $\sum_{i=1}^{4}u_{i}=1$, which are valid barycentric coordinates corresponding to the tetrahedron’s four vertices. In light of this, the tetrahedron can be viewed as a 3D region embedded in a 4D hyperspace. A sampled point $p$ on a camera ray is always located in the tetrahedron, but in order to compute the feature vector at $p$, we interpolate the features stored in the corners of a 4D voxel (see Figure 2). Further, the highest-level grid is subdivided into 16 4D voxels on the second level of the grid. But the 3D tetrahedron is embedded in only 5 of the 16 voxels. The rest of the voxels remains unused when we compute feature vectors of sampled ray points. See Figure 2 for visualization of this interpretation in 3D space.

We note that the unused voxels will not waste memory storage, because following Instant-NGP[MESK22], the feature vectors on grid nodes are stored in a hash table, and the hash table size is set on purpose much smaller than the grid size to “compress” feature vector storage. Meanwhile, considering that we need to maintain separate hash tables for each tetrahedron in the tetrahedron mesh of the Neural Impostor, when the number of tetrahedrons increases, the amount of information each tetrahedron actually needs to encode also decreases. Therefore, in practice, we reduce the hash table size of each tetrahedron according to the number of tetrahedrons in the mesh, in order to maintain the hash table size of the entire tetrahedron mesh comparable to that in the original Instant-NGP with a given level. For instance, consider a tetrahedron mesh with $T$ tetrahedrons and base-2 logarithmic hash table size $H$ , the corresponding local base-2 logarithmic hash table size in each tetrahedron would be $H^{\prime}=H/\lfloor(\log_{2}T+1)\rfloor$. The rest of the encoding process is similar to Instant-NGP. At each multiresolution grid level $l$, the feature vector $\boldsymbol{f}_{l}(\boldsymbol{x})$ at the sampling point $\boldsymbol{x}$ is obtained through four-dimensional linear interpolation, using the feature vectors at the $16$ corner points of the voxel containing $\boldsymbol{x}$. We concatenate feature vectors $\boldsymbol{f}_{l}(\boldsymbol{x})$ from all levels $l=1...L$ to form the feature vector $\boldsymbol{f}_{p}$ used for decoding. The decoding process of Neural Impostor mirrors that of NeRF[MST^∗20]. Given a density decoder, denoted $\mathsf{D}_{\sigma}$, and a radiance decoder $\mathsf{D}_{r}$, we have:

		$\displaystyle\mathsf{D}_{\sigma}:\mathbb{R}^{F\times L}\rightarrow\mathbb{R},\quad\sigma(p)=\mathsf{D}_{\sigma}(f_{p})$		(7)
		$\displaystyle\mathsf{D}_{r}:\mathbb{R}^{D^{2}+F\times L}\rightarrow\mathbb{R}^{3},\quad\boldsymbol{c}(p)=\mathsf{D}_{r}(f_{p}\oplus f_{d})$

herein, $p=\dot{o}+t\vec{d}$ denotes a sampled point along the ray, $f_{p}$ and $f_{d}$ represent the position feature and the direction feature of $p$ and $\oplus$ indicates feature concatenation.

During rendering, we need to calculate the color value $C(\boldsymbol{r})$ along the sampled camera ray $\boldsymbol{r}(t)=\dot{o}+t\cdot\vec{d}$, where $\dot{o}$ is the origin of the ray and $\vec{d}$ is the ray direction. Similar to the original NeRF model [MST^∗20], following the principles of Direct Volume Rendering (DVR), the color of the ray $C(\boldsymbol{r})$ can be estimated by integrating over a bounded distance interval $[t_{n},t_{f}]$:

$$C(\boldsymbol{r})=\int_{t_{n}}^{t_{f}}T(t)\sigma(\boldsymbol{r}(t))\boldsymbol{c}(\boldsymbol{r}(t),\vec{d})\operatorname{d}\!{t},$$

(8)

here, $T(t)=\exp\left(-\int_{t_{n}}^{t}\sigma(\boldsymbol{r}(s))\operatorname{d}\!{s}\right)$ represents the accumulated transmittance from $t_{n}$ to $t$ along the ray direction. $\sigma(\boldsymbol{r})$ is the volume density at location $\boldsymbol{r}(t)$; $\boldsymbol{c}(\boldsymbol{r}(t),\vec{d})$ is the radiance at $\boldsymbol{r}(t)$ in the direction $\vec{d}$.

The fundamental idea of NeRF[MST^∗20] is to represent $\sigma(\boldsymbol{r}(t))$ and $\boldsymbol{c}(\boldsymbol{r}(t),\vec{d})$ using neural networks. In our tetrahedron neural representation, the camera ray $\boldsymbol{r}$ intersects with a series of tetrahedron. During the deformation process, the volume of each tetrahedron might undergo significant changes. Under these circumstances, if we keep the Cartesian coordinate-based integral method, the final color and transparency of the ray would directly correlate with the integral distance. Consequently, any changes in the integral distance due to the deformation of the tetrahedron would also affect the output color.

To dissociate the color of the ray from the integral distance, we leverage the sampling method discussed in Section 3.2. We rewrite Equation (8) as a sum of the integrals for each tetrahedron, while converting the integral distance into a distance representation in barycentric coordinates. This ensures that the color of the ray remains unaffected, even when the volume of the tetrahedron experiences significant deformation. As illustrated in Figure 1, we denote the $K$ tetrahedron that the ray passes through during propagation as $t_{1},t_{1},\cdots,t_{K}$. Each tetrahedron intersects with the ray at entry point $p_{0}$ and exit point $p_{1}$, with corresponding barycentric coordinates $\Lambda_{t}^{0}$ and $\Lambda_{t}^{1}$, respectively. We use the Manhattan distance between $\Lambda_{t}^{0}$ and $\Lambda_{t}^{1}$ as the integral distance, and recast Equation (8) as a sum of integrals over each tetrahedron:

$$C(\boldsymbol{r})=\sum_{i=1}^{K}\int_{\Lambda_{t_{i}}^{0}}^{\Lambda_{t_{i}}^{1}}T(\Lambda_{t_{i}}^{p})\sigma(\Lambda_{t_{i}}^{p})\boldsymbol{c}(p,\vec{d})\ d{\Lambda_{t_{i}}^{p}}$$

(9)

Here, $\Lambda_{t_{i}}^{p}$ represents the barycentric coordinates of the sampling point $p$ within the tetrahedron $t_{i}$. Due to the linearity of the barycentric coordinates, $\Lambda_{t_{i}}^{p}$ can be determined by the barycentric coordinates of the entry and exit points, $\Lambda_{t}^{0}$ and $\Lambda_{t}^{1}$, respectively, as well as the interpolation weight $\alpha$ as defined in Equation (3).

The radiance in a scene typically carries a higher frequency of information than the spatial density. Therefore, the radiance decoder in the process of constructing a neural radiance field usually exhibits more complexity than the spatial density decoder. Moreover, the forward process of a MLP is more costly in terms of time efficiency compared to encoding and rendering. To further optimize the rendering efficiency, we first decode the position features of the sample points into density values using a few density decoders $\mathsf{D}_{\sigma}$. In contrast to previous rendering schemes that decode before integration for radiance, we propose a rendering scheme that integrates before decoding. Specifically, we first integrate the position features of the sampling points and ray transmittance onto its belonging tetrahedron surfaces, then alpha composite them together into a final appearance feature for each ray. Subsequently, the composited appearance feature are combined with the ray directional feature for radiance decoding, as such we only need to computer the appearance decoder once for each ray:

$$C(\boldsymbol{r})=\mathsf{D}_{r}\Bigl{[}\sum_{i=1}^{K}\underbrace{\int_{\Lambda_{t_{i}}^{0}}^{\Lambda_{t_{i}}^{1}}T(\Lambda_{t_{i}}^{p})\sigma(\Lambda_{t_{i}}^{p})\mathsf{E}_{p}^{t_{i}}(\Lambda_{t_{i}}^{p})\ d{\Lambda_{t_{i}}^{p}}}_{\textit{integrated positional feature}}\ \oplus\ \mathsf{E}_{d}(\vec{d})\Bigr{]}$$

(10)

here, $\mathsf{D}_{r}$ is the radiance decoder, and $\mathsf{E}_{p},\mathsf{E}_{d}$ are the encoders for the barycentric coordinates and ray direction, respectively. The density values of the sample points are obtained by decoding the barycentric coordinates according to Equation (7). This "integration-before-decoding" rendering approach effectively pre-bakes the scalar field of the encoding space onto the surface of each tetrahedron. As each ray can define a unique surface point for each tetrahedron, this "integration-before-decoding" approach allows us to rasterize the tetrahedron in the space layer by layer and project them onto the imaging plane to further accelerate the rendering process during inference.

While the "integration-before-decoding" rendering approach can accelerate the rendering process during inference, in the barycentric coordinate-based encoding method, we also need to determine the tetrahedron to which each sample point belongs during the training process. For a 1080p image, the forward process of training requires conducting a barycentric coordinate test for approximately $1080\times 1080\approx 10^{8}$ points. Traditional tests based on matrix multiplication would require us to compute the determinant of a $N_{rays}\times N_{points}\times 4\times 3$ matrix, which is unfeasible for complex tetrahedron meshes in one forward pass. Building upon the work of Wald et al. (2019) [WUM^∗19], we consider the triangles that make up the tetrahedron as the testing units during the rendering process. For non-self-intersecting tetrahedron meshes, the concept of a point being "in front of a triangle" or "behind a triangle" can be uniquely determined by the winding order of the triangle vertices. In their method, determining to which tetrahedron a point belongs is equivalent to casting a ray from that point in a random direction, evaluating the winding order at the first intersection point, and checking on which side of the triangle the sample lies. Since each triangle belongs to at most two tetrahedron, this winding order-based testing approach can yield a unique tetrahedron index for any point in space. When hardware acceleration is available, the triangle-based query operation is also very efficient in terms of memory and computation.

We extend the method of [WUM^∗19] to the framework of ray tracing to support the sampling process in Section 3.2. Specifically, in the context of ray tracing, we can perform stepwise sampling along the direction of the ray. Unlike the separate testing for each point in the method of Wald et al. (2019) [WUM^∗19], we only need to determine the entry and exit points of the tetrahedron that each ray passes through. Depending on the arrangement of tetrahedron in space, the progression of the ray may encounter three situations shown in Figure 3, namely adjacent/non-adjacent tetrahedron and intersecting tetrahedron. We adopt different stepping strategies for these three situations. As shown in Figure 3, for adjacent tetrahedron that share one triangle faces, we take the entry point as the origin of the ray and step to determine the exit point. After sampling, the entry point is updated to the exit point. For separated tetrahedron, we connect them at runtime. That is, if the sample point is at position $p_{2}$ in Figure 3, we first update it to $p_{3}$ before sampling. For intersecting tetrahedron, we use an aggressive sampling approach, i.e., for a sample point $p_{4}$ that belongs to both tetrahedron $A$ and $B$, we add its position in both $A$ and $B$ to the sampling set.

Shape transformation and deformation are the most fundamental geometric editing operations. Fundamentally, shape transformation and deformation reveal geometric invariance under continuous mappings, which can naturally be satisfied by the hybrid modeling of Neural Impostor. Thanks to the centroid coordinate encoding method described in Section 3.1, in Neural Impostor any spatial transformation based on tetrahedron vertices can be naturally transformed into the implicit field. For example, in Figure 4, the input shape morphs from the original shape $M$ on the left to the head rescaled scaled shape $M^{\prime}$ on the right. Recalling equation 2, the sampled points in the deformed tetrahedron mesh $M^{\prime}$ are given by:

$$t,\Lambda_{t}^{p}\leftarrow\mathsf{S}(p,M^{\prime}),\textit{ where }p\in\mathbb{R}^{3}\text{{ and }}t\in M^{\prime}$$

(11)

In this case, both the sampling spaces before and after deformation are defined in the Cartesian space of the scene. When the tetrahedron vertices move, the implicit neural field inside the deformed tetrahedron remain unchanged under the barycentric coordinate. Thus, we can adopt the same sampling, encoding, and rendering methods as the original process without any additional transformations. Moreover, as this process only involves updates to the tetrahedron mesh vertices without calculating the transformations on sampling points, the deformation process in Neural Impostor is more efficient than approaches based on bending rays[XH22, PYL^∗22, GKE^∗22, YSL^∗22].

Compared to continuous shape morphing, topological operations pose more challenges. When the mesh topology changes, the number and connectivity of the original tetrahedron are altered. However, in Neural Impostor the radiance field is strictly tied to the tetrahedron topology. This requires an effective mapping scheme to transfer the radiance field baked into the original tetrahedron mesh to the new tetrahedron structure.

To achieve this goal, we propose a local retraining strategy to synthesize neural implicit fields from existing neural implicit fields with different explicit tetrahedral mesh. When remeshing happens, rather than performing a global retraining, we select local tetrahedron where the topology changes for retraining while keeping the other tetrahedron unchanged. Besides, to accelerate convergence, we initialize the updated multigrid hash table based on feature-only supervision shown inFigure 5 (c). Then follow a joint optimization on both the feature hash table and the MLP weights. As shown in Figure 5, during the retraining stage, our input is the pre-trained Neural Impostor which includes its tetrahedron mesh and the implicit neural field encoded in each tetrahedron illustrated in Figure 5 (b), as well as the explicit tetrahedron mesh after the topological transformation as shown in Figure 5 (a). The goal of local retraining is to transfer the radiance field defined on the original tetrahedrons to the new radiance field on the remeshed tetrahedral mesh. The retraining process consists of two stages. In the first stage, as shown in Figure 5(c), we randomly sample points inside (solid dots in Figure 5 (a)) and outside (hollow dots in Figure 5 (a)) of the new mesh to optimize the hash table. In the second stage (Figure 5 (d)), we use ray tracing to locate the pixel positions in that region and select effective rays to obtain pixel colors as further supervision according to the rendering process in Equation (10), this improving the accuracy of the rendered results after retraining. Specifically, let $(M,h,\mathsf{D})$ represent the tetrahedron mesh, hash encoding table, and decoder of the Neural Impostor before retraining, and let $(M^{\prime},h^{\prime},\mathsf{D}^{\prime})$ represent the reconstructed tetrahedron mesh and its to-be-trained hash encoding table and decoder. Let $\mathsf{R}$ and $\mathsf{E}_{d}$ represent the renderer and ray direction encoder used in the second stage training. The training process can be formalized as follows:

	Stage 1:	$\displaystyle\mathcal{L}_{hash}=\|h(\Lambda_{p})-h^{\prime}(\Lambda_{p})\|^{1},\textit{ where }p\in M\cup M^{\prime}.$		(12)
	Stage 2:	$\displaystyle\mathcal{L}_{render}=\Bigg{\|}\mathsf{R}\circ\mathsf{D}\bigl{[}h(\Lambda_{p})\oplus\mathsf{E}_{d}(\vec{d})\bigr{]}-\mathsf{R}\circ\mathsf{D^{\prime}}\bigl{[}h^{\prime}(\Lambda_{p})\oplus\mathsf{E}_{d}(\vec{d})\bigr{]}\Bigg{\|}^{2},$
		$\displaystyle\textit{where }p\in\dot{o}+t\vec{d}.$

where $\circ$ denotes function composition and $\oplus$ represents feature concatenation.

Besides the operations on the explicit part in the tetrahedral neural representation, inter-operations between different neural radiance fields are essential to certain cases, such as Boolean operations. To merge or subtract two neural radiance fields, we need to define boolean status on the implicit fields. Luckily, its much easier to define the boolean status on the implicit field(such as NeRF) than the explicit geometry(such as mesh). For explicit geometry, the boolean status is defined by the in/out status of the sampling point, which is not well defined for non-watertight geometry, and usually hard to compute. On the other hand, the implicit field is indexed with positional parameters in a universal encoding space $G$, which allows us to define the boolean status based on the scalar values interpreted for each sampling point. Here we use a straightforward filtering algorithm to define the boolean status $\mathcal{B}(\lambda)$ based on simple thresholds:

$$\mathcal{B}(\lambda)=\begin{cases}1,&\text{if }\mathcal{D}_{\sigma}(\lambda)>\epsilon\\ 0,&\text{otherwise}\end{cases}$$

(13)

Therefore, the boolean field $\mathcal{B}$ is a binary function of the density field $\mathcal{D}$. Based on the boolean field, we can define the basic boolean operations between neural radiance fields $\mathcal{B}_{i}$ and $\mathcal{B}_{j}$ as follows:

		$\displaystyle\mathcal{B}_{i}\cup\mathcal{B}_{j}=\mathcal{B}_{i}+\mathcal{B}_{j}-\mathcal{B}_{i}\cdot\mathcal{B}_{j}$		(14)
		$\displaystyle\mathcal{B}_{i}\cap\mathcal{B}_{j}=\mathcal{B}_{i}\cdot\mathcal{B}_{j}$
		$\displaystyle\mathcal{B}_{i}\setminus\mathcal{B}_{j}=\mathcal{B}_{i}-\mathcal{B}_{i}\cdot\mathcal{B}_{j}$
		$\displaystyle\mathcal{B}_{i}\oplus\mathcal{B}_{j}=\mathcal{B}_{i}+\mathcal{B}_{j}(mod2)$

Then we mask out the corresponding region of the objects and do the local re-training by using the result boolean field as binary weights for the volumetric rendering process.

$\displaystyle C_{k}=\mathsf{D}_{r}(\underbrace{\int_{t_{0}}^{t_{1}}T_{tet}(\lambda_{t})\underbrace{\mathcal{B}(\lambda_{t})}_{\textbf{{boolean}}}\underbrace{\mathsf{D}_{\sigma}(\mathsf{E}_{p}(\lambda_{t}))}_{\textit{density}}\cdot\underbrace{\mathsf{E}_{p}(\lambda_{t})}_{\textit{pos feature}}dt}_{\textit{accumlated feature}}\oplus\underbrace{\mathsf{E}_{d}(\vec{d})}_{\textit{dir feature}})$

(15)

As shown in Figure 6, through implicit boolean operations, we can add constructive pattern details to a pre-trained Neural Impostor model. Please note that the boolean field determines the presence of the density field, but the color of the model is interpreted based on the selection of the radiance field. By the way, boolean operations can be accelerated by first checking the boolean state of tetrahedron and then performing implicit boolean operations on the selected tetrahedron. By default, boolean operations are binary selective operations on implicit fields. As a natural extension, selective blending can be achieved using the similar strategy. Instead of keep or drop a radiance field, we can blend or perform any mutual operations on two radiance fields. For example, as shown in Figure 6 (c), within the region defined by the pattern, we multiply the color given by the pattern with the density field representing fur in the original Neural Impostor resulting in a fur model with color. For such cases, local retraining is accomplished by using the second-stage blending perspective-related appearance as the target after determining the tetrahedron covered by the pattern:

	$\displaystyle C_{k}=$	$\displaystyle\mathsf{D}_{r}^{i}(\int_{t_{0}}^{t_{1}}T_{tet}^{i}(\lambda_{t}^{i})\mathcal{B^{i}}(\lambda_{t}^{i})\mathsf{D}_{\sigma}^{i}(\mathsf{E}_{p}^{i}(\lambda_{t}^{i}))\cdot\mathsf{E}_{p}^{i}(\lambda_{t}^{i})dt\oplus\mathsf{E}_{d}^{i}(\vec{d^{i}}))$		(3-16)
	$\displaystyle+$	$\displaystyle\mathsf{D}_{r}^{j}(\int_{t_{0}}^{t_{1}}T_{tet}^{j}(\lambda_{t}^{j})\mathcal{B^{j}}(\lambda_{t}^{j})\mathsf{D}_{\sigma}^{j}(\mathsf{E}_{p}^{j}(\lambda_{t}^{j}))\cdot\mathsf{E}_{p}^{j}(\lambda_{t}^{j})dt\oplus\mathsf{E}_{d}^{j}(\vec{d^{j}}))$

As mentioned earlier, we analyze the modeling capability of Neural Impostor from the perspectives of modeling, rendering efficiency, and continuity in the deformation process. In this section, we focus on evaluating the reconstruction quality. Specifically, using the implementation of Instant-NGP in nerfstudio[TWN^∗23] (method "NGP-Bounded" in Table 2) as a baseline, we first replace the Cartesian coordinate-based encoding part in the original model with barycentric coordinate encoding while retaining the sampling method based on occupancy fields (method "BaryEnc" in Table 2). Then, we replace the sampling method based on occupancy fields with barycentric coordinate-based sampling (method "BarySpl" in Table 2). Finally, we integrate the "early integration" rendering approach from Section 3.4 into the model based on the barycentric coordinate sampling method (method "FeatInt" in Table 2). We also provide the results of rendering after baking onto the tetrahedral surface and using rasterization for rendering (method Baking+Rast in Table 2). Among them, we compare the PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) under various rendering methods for 7 instances in the nerf-synthetic dataset and 5 instances in the Plush Toy dataset (including 3 real-captured instances and 2 rendered instances). PSNR measures the quality of an image by computing the mean squared error between the original image and the processed (compressed or reconstructed) image. The higher the PSNR value, the smaller the difference between the processed image and the original image, indicating better quality. SSIM measures the similarity between two images by considering their luminance, contrast, and structural information. It is a more accurate reflection of image quality compared to traditional metrics like PSNR, as it aligns better with human visual perception. We provide visual comparisons of the reconstruction results on the nerf-synthetic and plush toy datasets in Figure 8 and Figure 7, as well as their quantitative results in Table 2. The quantitative analysis on the nerf-synthetic dataset shows that, without introducing the early integration rendering approach (Section 3.4), using the barycentric coordinate encoding in the modeling process (method "BaryEnc") achieves comparable quality to the original Instant-NGP [MESK22], with a PSNR value of 33.134 for barycentric coordinate encoding compared to 33.272 for the original Instant-NGP encoding. Furthermore, since barycentric coordinate encoding is not limited by resolution, it can achieve good reconstruction quality (PSNR = $32.915$) in complex scenes such as plush toys when combined with barycentric coordinate-based sampling, surpassing the quality of the original Instant-NGP (PSNR = $32.708$). When the early integration (Section 3.4) approach is introduced, although rendering efficiency improves, there is a slight decrease in modeling quality due to the texture features of each tetrahedron being baked onto the tetrahedral surface. However, after baking, we can use rasterization for rendering, resulting in a significant improvement in rendering efficiency (from $30+$ FPS before baking to $100+$ FPS after baking). Therefore, in scenarios with high interactivity demands, we can choose to use the early integration (Section 3.4) rendering approach to improve interactive performance.

The proposed hybrid representation relies on an explicit tetrahedron mesh as a proxy of the neural implicit field. The quality and density of tetrahedron mesh do impact the encoding of the neural radiance field. However, these factors only have subtle effects on the reconstruction quality. Figure 9 examines the impact of different tetrahedral mesh quality and density on the reconstruction quality of neural impostors in static scene reconstruction. We tests 7 different tetrahedron proxies with similar implicit field sizes (i.e. sum of hash table size of all tetrahedrons) on the fluffy ball example by varying the tetrahedron density and envelope shape. The similar PSNR of all tests indicates that the selection of tetrahedron proxies has an insignificant influence on the quality of the implicit field.

Different from evaluating the reconstruction quality, there is no well-defined metric or reference ground truth for the edited radiance field. Therefore, we use the rendering of the edited Neural Impostor as a qualitative evaluation(please check Section 6.1) and calculate the LPIPS (Learned Perceptual Image Patch Similarity) between the edited Neural Impostor to the reference rendering as the quantitative evaluation. Here, we analyzed the rendering quality during simulation. Specifically, we use the Houdini simulation engine to create dynamic meshes for fracture and deformation effects for the testing scenes. We then perform animation simulations using the ray bending algorithm in NeRF-Editing[YSL^∗22] with two-stage sampling (Ray Bending in Table 3), the algorithm based on occupancy field sampling combined with barycentric coordinate encoding (Occupancy Field in Table 3), and the barycentric-based sampling algorithm in Section 3.2 (Barycentric Sampling in Table 3). Since we cannot obtain real multi-view data as a reference during the simulation process, and the deformations generated during the simulation can be negligible with a sufficiently high frame rate, we calculate the LPIPS between each pair of frames as a measure of animation simulation quality. From the comparison of LPIPS between each pair of frames in Table 3, it can be observed that our barycentric sampling method better maintains the appearance stability during the deformation process.

Soft body deformation algorithms are used in computer graphics to simulate the motion and deformation of soft, flexible materials such as cloth, rubber, and muscles. These algorithms combine physics simulation with numerical methods to calculate the motion and deformation of soft bodies under external forces. The basic idea of soft body deformation is to discretize the object into many small elements, and solve for the displacement for each node in the elements. The motion and deformation of each element are calculated based on the forces acting on it, such as gravity, collision forces, tension, and compression forces. One common method of soft body deformation is using finite element analysis (FEA), which divides the body into small triangles or tetrahedral elements. Each element is assigned a set of physical properties, such as stiffness and density, which are used to calculate the motion and deformation of the element under external forces. Other soft body deformation algorithms include particle-spring systems, which model the body as an interconnected network of springs, and position-based dynamics, which use simplified equations to simulate the motion and deformation of the body. By using the same discretized tetrahedron mesh, we can achieve physical simulation directly on Neural Impostors. Figure 10 shows an example of soft body simulation based on Neural Impostor. In this example, we use a furball with a complex appearance but relatively simple geometry (consisting of 204 vertices and 600 tetrahedra) as the object of simulation. Three significantly different viewpoints and five dynamic frames are extracted and displayed. From the figure, it can be observed that the Neural Impostor can achieve continuous simulation with large deformations while maintaining the quality of the volumetric appearance. Furthermore, in Figure 11, we demonstrate continuous soft body transformations based on rendered data and real-shot data. Plush Spot represents the rendered data, while Plush Bear represents the real-shot data. For more examples, please refer to the supplementary video material.

In addition to soft body simulation, plastic fracture simulation is a technique used in computer graphics to simulate the destruction of buildings, vehicles, and other structures. In plastic fracture, each part of the object is treated as a separate plastic element that can be influenced by external forces such as explosions or collisions. These plastic elements are simulated using physics-based algorithms to calculate their motion and deformation over time. One common method for simulating plastic fracture is to use Voronoi splitting to break the object into smaller fragments. These fragments are then simulated using a physics engine to calculate their motion in the scene and collisions with other objects. Other techniques used for plastic fracture include finite element analysis (FEA) and dynamic fracture modeling, which can produce more detailed and realistic simulations of object destruction. Figure 12 illustrates two examples of plastic fracture simulation on Neural Impostors. Similar to soft body simulation, we utilize the Houdini engine with the original tetrahedral mesh as a proxy for the plastic fracture simulation.

In addition to its robust support for vertex-based animation, as shown in Figure 6, the Neural Impostor algorithm further enables Boolean appearance editing (e.g., baking pattern colors onto an existing Neural Impostor model as shown in Figure 6(d)) and geometry editing (e.g., sculpting the original Neural Impostor as shown in Figure 6(c)) through local retraining operations described in Section 4. By using explicit tetrahedral meshes as proxies, we can effortlessly combine multiple Neural Impostor models to create new scenes, offering the possibility of using Neural Impostor as a novel modeling primitive. Figure 13 illustrates the process of creating a new Neural Impostor asset from material spheres and three existing Neural Impostor models as shown in Figure 13(a) through appearance mapping (snowman body), composition (hat, nose, eyes, and other decorations), and geometry editing (snowman arms). Specifically, starting from material spheres representing the appearance of a snowball and wooden arms, we transfer the appearance information represented by their materials to the snowman body and arms, represented by a simple spherical geometry, through local retraining. Additionally, we elongate the arm vertices along the axis to assemble the snowman body. By combining the constructed body parts with Neural Impostor representing the hat and nose, we create a complete snowman model as shown in Figure 13(b). It is worth noting that, apart from the retraining step involved in appearance mapping, this modeling process aligns completely with traditional 3D modeling software workflows. After modeling, the snowman remains a new Neural Impostor model, allowing seamless integration with subsequent animation simulations. Figure 13(c) illustrates the process of soft body simulation applied to the snowman model.

The above example is just one of many possibilities. In practical applications, the Neural Impostor technology can be used to create various 3D models, including animals, buildings, vehicles, and more. Additionally, inspired by the construction of neural radiation fields from multi-view images, artists can start appearance modeling directly from real captured images, simplifying the material design process in traditional modeling. In a nutshell, the Neural Impostor technology provides a brand new workflow for authoring complex models with high quality volumetric appearance.