Towards Efficient Neural Scene Graphs
by Learning Consistency Fields

Being a long standing issue in computer vision, various approaches have been proposed for the task of novel view synthesis, including those using point clouds, discretized voxel grids and textured mesh [Fan et al.(2017)Fan, Su, and Guibas, Yan et al.(2016)Yan, Yang, Yumer, Guo, and Lee, Sitzmann et al.(2019)Sitzmann, Thies, Heide, Nießner, Wetzstein, and Zollhofer]. Neural Radiance Fields (NeRF) [Mildenhall et al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng] represents a static scene by evaluating a volumetric density (transparency) and a view-dependent color. The object’s details can be expressed at a low storage cost via an implicit 5D (spatial location ($x,y,z$) and viewing direction ($\theta,\phi$)) function implemented with a multi-layer perceptron (MLP).

A similar idea is applied to understand a video scene. Unlike a still image, a video has an additional temporal-axis, and objects in the scene may be static or dynamic (moving) across time. Neural Scene Graphs (NSG) for dynamic scenes [Ost et al.(2021)Ost, Mannan, Thuerey, Knodt, and Heide] provides a considerable potential on various applications by enabling the understanding of a complex scene with dynamic multi-objects, which has been tricky to model. Also, NSG extends the task of novel view synthesis to novel scene manipulation, allowing spatial rearrangements of dynamic objects in the scene (e.g., the second row in Fig. 6). However, direct application of NSG is still not practical due to heavy computational overhead at inference. Along with other NeRF variants, NSG also suffers from expensive ray marching for all sampled queries for rendering. Moreover, since NSG repeats this procedure for each image frame independently, the total inference cost scales with the spatio-temporal resolution of the video. Observing this limitation, a natural question arises: how can we efficiently reduce this computational overhead?

We start from an idea that most objects in a video do not significantly change in adjacent frames. In other words, there is much redundancy in the representation of the scene across image frames as visual semantics are largely consistent. Reusing the previously-computed representations, instead of repeating computations for similar queries again, would be a plausible way to significantly improve efficiency.

We conduct a simple experiment to measure redundancy across frames. We divide an object’s bounding box into uniform spatial bins and assign queries $(x,y,z)$ to each bin, storing the estimated color and density values. Fig. 2(a) shows the values across all frames for two selected bins [A] and [B]. We found that the color and density values from the consecutive frames hardly or only slightly change within a limited range. Fig. 2(b) shows the ratio of all bins whose values change less than a threshold $\epsilon$ in two consecutive frames. As the range of radiance and density are within $[0,1]$, the possible threshold for $\epsilon$ is also within $[0,1]$. We note that redundancy is already close to 70% at $\epsilon\approx 0.04$. To facilitate better understanding of $\epsilon$, we generate images with random salt-and-pepper noise of $\epsilon$ = $\{0.04,0.18\}$ for all pixels. As shown in Fig. 2(c), the image with $\epsilon=0.04$ noise is almost identical to the original image maintaining high PSNR. The image with $\epsilon=0.18$ noise (90% redundancy in (b)) still well-preserves the overall scene. This experiment indicates that most of bins share highly similar values across frames. Hence, actively leveraging this redundancy is not only beneficial but in fact necessary to guide our model to learn dynamic scenes more efficiently.

To this end, we propose to identify visual components that are consistent across frames, and reuse them for improved efficiency. Fig. 1 shows the rendered image using our method with 85% less queries without significantly degrading rendering quality. In this paper, we propose Consistency-Field-based NSG (CF-NSG), which effectively reduces redundant computation across frames for more efficient rendering. We first review NSG in Sec. 2, and analyze a critical factor that hinders the feature reusing mechanism in Sec. 3. Then, we introduce our CF-NSG in Sec. 4. We empirically demonstrate that our model greatly improves efficiency with no distinguishable compromise in image quality in Sec. 5.

Neural Radiance Fields (NeRF) [Mildenhall et al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng] captures 3D representations of static objects or scenes. The implicit representation is encoded into an MLP, which maps a 3D spatial location $\textbf{{x}}=(x,y,z)$ to its volumetric density $\sigma$, combined with a viewing direction $\textbf{{d}}=(\theta,\phi)$ to an emitted color $\textbf{{c}}=(r,g,b)$. The color of the pixel $C(\textbf{{r}})$ along a camera ray r is estimated by accumulating the color and transmittance of $N$ sampled points along the ray:

$$\footnotesize\hat{C}(\textbf{{r}})=\sum_{i=0}^{N-1}T_{i}\left(1-\exp(-\sigma_{i}\delta_{i})\right)\textbf{{c}}_{i},\quad T_{i}=\exp\Bigg{(}-\sum_{j=0}^{i-1}\sigma_{j}\delta_{j}\Bigg{)}$$

(1)

where $\delta_{i}$ is the distance between adjacent sampled points. The NeRF network is optimized to reduce the $L_{2}$ distance between the estimated colors $\hat{C}(\textbf{{r}})$ for a random a batch of rays $\mathcal{R}$ and their ground truth (GT); that is, $\mathcal{L}=\sum_{\textbf{{r}}\in\mathcal{R}}\|C(\textbf{{r}})-\hat{C}(\textbf{{r}})\|_{2}^{2}$.

The overall process can be expressed by

$$\small\mathcal{F}_{\theta_{bg}}:(\textbf{{x}},\textbf{{d}})\rightarrow(\textbf{{r}},\textbf{{g}},\textbf{{b}},\sigma),\quad\quad\mathcal{F}_{\theta_{c}}:(\textbf{{x}}_{o},\textbf{{d}}_{o},\textbf{{p}}_{o},\textbf{{l}}_{o})\rightarrow(\textbf{{r}},\textbf{{g}},\textbf{{b}},\sigma),$$

(2)

where $\theta_{bg}$, $\theta_{c}$ are respective weights for static and dynamic representation models. Here, $c$ denotes each class of dynamic objects, where $c\in\{c_{1},\cdots,c_{n}\}$ for $n$ classes. In the dynamic model, the set of dynamic objects belonging to class $c$ ($\mathcal{O}_{c}$) shares weights $\theta_{c}$, while latent vector $\textbf{{l}}_{o}$ is uniquely learned for an individual object $o\in\mathcal{O}_{c}$. $\textbf{{p}}_{o}$ is the global location of object $o$ in the scene. Each $\textbf{{x}}_{o}$ and $\textbf{{d}}_{o}$ refer to the spatial location and the viewing direction, respectively, where the corresponding color values are desired. Note that $\textbf{{x}}_{o}\in[-1,1]^{3}$ and $\textbf{{d}}_{o}$ are in the canonical coordinates of object $o$, different from x and d in global coordinates. Outputs $(r,g,b,\sigma)$ from the dynamic models are mapped from canonical coordinates to global coordinates and integrated simultaneously with the outputs from the static model along the rays, composing a scene similarly as in [Niemeyer and Geiger(2021)].

NSG extends the realm of NeRF-based applications to videos with multiple dynamic objects. However, computation cost at inference is still prohibitive, as image rendering requires ray marching operations per frame. Thus, we propose a feature-reusing framework that stores redundant features across frames during training, then at inference reuses them instead of going through a full forward pass. Assuming the features in the nearby voxels are similar, we create quantized memory bins in the coordinates as follows. For the dynamic models ($\mathcal{F}_{\theta_{c}}$ in Eq. (2)), we quantize the object-specific 3D coordinates (within the object’s bounding box) into spatial bins, similarly as in the experiment in Fig. 2. For the static model ($\mathcal{F}_{\theta_{bg}}$ in Eq. (2)), following NSG, we define radiance fields and grid memory bins on a set of 2D planes instead of volume. If a query is assigned to the bin whose value is expected not to significantly change across frames, we reuse previously computed results from the bin.

Fig. 3 illustrates the qualitative results. Fig. 3(a) shows an example of a reproduced dynamic object, a vehicle. It clearly shows that reusing RGB color values generates abnormalities, e.g., blended colors at the rear of the grey car. The naive reusing approach is also imperfect for reproducing a static background. As seen in Fig. 3(b), the street lamp on the right or trees in the back (yellow circled) cannot be pinned down, displaying ghost effects. This indicates the reusing decision was made erroneously, reusing the features that are significantly changes across the frames.

First, we define two distinct properties that are canonical to the object (e.g., intrinsic color [Zhang et al.(2019)Zhang, Wang, and Zhang] or shape) and those from external factors. The former should be maintained consistent across frames regardless of the object’s current location or viewing direction, while the latter may change depending on the spatio-temporal environment. Here, spatio-temporal consistency incorporates both actual time change or movement of an object (temporal change) and the viewpoint change coming from the use of stereo-cameras on different locations (spatial change). Therefore, we aim to reuse features representing canonical properties.

However, we hypothesize that NSG fails to completely disentangle features intrinsic to each object and those affected by its transient environment. Since RGB color values are produced by both object-internal and external factors, supervision from RGB color (GT pixel values) alone would be unable to instruct about canonical properties of the object. To support our hypothesis, we conduct an experiment shown in Fig. 4. The first row shows two objects, a white car ($o$) and a truck behind it ($o^{\prime}$) in the training set, whose bounding boxes $\textbf{{p}}_{o}$ and $\textbf{{p}}_{o^{\prime}}$ are marked in red and white, respectively. In the second row, we render the white car ($o$) at intermediate locations between $\textbf{{p}}_{o}$ and $\textbf{{p}}_{o^{\prime}}$, using the NSG representation of the car ($\textbf{{l}}_{o}$). Even though we only change its physical location, we observe that canonical properties of the object, e.g., its color and shape of the tail lights, are significantly contaminated depending on its location. This result reveals that the object-intrinsic and environmental representations are severely entangled in NSG. That is, the object-intrinsic representation learned by NSG is heavily affected by transient factors, such as the viewing direction or its location.

To improve the efficiency of NSG by the feature-reusing framework while maintaining the image quality, our observations from Sec. 3 imply two conditions: 1) a solid criterion should be used to determine which bins are reusable without hurting the rendering quality, and 2) the reused features should be able to properly represent the characteristics that is intrinsic to each object. We propose the Consistency-Field-based NSG (CF-NSG), illustrated in Sec. 4.1 and Fig. 5, satisfying these two conditions and achieving disentangled representations. By the term consistency, we refer to the characteristics of the query that strongly show canonical and consistent properties of the object across frames. We call our method Consistency-Field since CF-NSG reformulates the radiance fields to additionally measure the consistency of each query across frames. We also discuss how we further boost the efficiency and adapt our final model to practical settings under limited memory in Sec. 4.2.

We also show that our method indeed learns disentangled representations regarding consistency. We conduct a similar experiment as in Sec. 3 for CF-NSG, and show the result in the third row of Fig. 4. Our CF-NSG relatively well preserves object-intrinsic properties regardless of the object’s location. Making use of disentangled representations, we also can apply CF-NSG to various scene compositions in a more stable manner (see Supp. Material).

Recently, the advancement of implicit or neural representations has enabled researchers to achieve photo-realistic views [Jiang et al.(2020)Jiang, Sud, Makadia, Huang, Nießner, Funkhouser, et al., Mescheder et al.(2019)Mescheder, Oechsle, Niemeyer, Nowozin, and Geiger, Niemeyer et al.(2020)Niemeyer, Mescheder, Oechsle, and Geiger]. To suggest a better representation for over-smoothed renderings of the existing methods, Mildenhall et al. [Mildenhall et al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng] introduced Neural Radiance Fields (NeRF). However, training and rendering processes based on neural representations often require time-consuming ray marching. To improve efficiency of NeRF-based models, some research has introduced more efficient data structures e.g., caching [Hedman et al.(2021)Hedman, Srinivasan, Mildenhall, Barron, and Debevec, Garbin et al.(2021)Garbin, Kowalski, Johnson, Shotton, and Valentin, Müller et al.(2022)Müller, Evans, Schied, and Keller], visual hull [Kondo et al.(2021)Kondo, Ikeda, Tagliasacchi, Matsuo, Ochiai, and Gu], sparse voxel [Liu et al.(2020)Liu, Gu, Lin, Chua, and Theobalt], view-dependenet multiplane image [Wizadwongsa et al.(2021)Wizadwongsa, Phongthawee, Yenphraphai, and Suwajanakorn] and octree [Yu et al.(2021)Yu, Li, Tancik, Li, Ng, and Kanazawa].

Another stream of research introduced representations of complex and dynamic scenes [Li et al.(2021)Li, Niklaus, Snavely, and Wang, Pumarola et al.(2021)Pumarola, Corona, Pons-Moll, and Moreno-Noguer, Park et al.(2021)Park, Sinha, Hedman, Barron, Bouaziz, Goldman, Martin-Brualla, and Seitz]. To clarify the difference between static and dynamic scene representations, the latter involves movement in both the camera and the scene. Hence, it is important to consider interactions with the scene such as global illumination to determine the appearance of a dynamic object. Due to this difference, more factors need to be considered to efficiently render the dynamic scenes besides simply modeling each canonical space of dynamic object as an efficient radiance fields for static scenes i.e., directly applying  [Müller et al.(2022)Müller, Evans, Schied, and Keller, Wizadwongsa et al.(2021)Wizadwongsa, Phongthawee, Yenphraphai, and Suwajanakorn, Kondo et al.(2021)Kondo, Ikeda, Tagliasacchi, Matsuo, Ochiai, and Gu, Hedman et al.(2021)Hedman, Srinivasan, Mildenhall, Barron, and Debevec, Yu et al.(2021)Yu, Li, Tancik, Li, Ng, and Kanazawa, Liu et al.(2020)Liu, Gu, Lin, Chua, and Theobalt, Garbin et al.(2021)Garbin, Kowalski, Johnson, Shotton, and Valentin].

We propose CF-NSG, a novel framework for representing multi-object dynamic scenes efficiently by utilizing a feature-reusing framework based on consistency-fields. CF-NSG is able to render images using only 15% of the original number of queries with little compromise to the image quality. Also CF-NSG enjoys more extended novel scene manipulation, taking advantage of disentangled representation with respect to spatio-temporal consistency.

This work was supported by the New Faculty Startup Fund from Seoul National University and by National Research Foundation (NRF) grant (No. 2021H1D3A2A03038607/15%, 2022R1C1C1010627/15%) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant (No. 2022-0-00264/10%, 2021-0-01778/10%, 2022-0-00320 /25%, No.2022-0-00953/25%) funded by the Korea government (MSIT).