Spatiotemporal Texture Reconstruction for Dynamic Objects
Using a Single RGB-D Camera

3D reconstruction systems for dynamic objects can be classified into two categories: template-based and templateless. A representative templateless work is DynamicFusion [NFS15], which shows real-time non-rigid reconstruction using a single depth camera. VolumeDeform [IZN^∗16] considers a correspondence term made from SIFT feature matching to improve motion tracking. BodyFusion [YGX^∗17], which specializes in human body, exploits human skeleton information for surface tracking and fusion.

Template-based non-rigid reconstruction assumes that a template model for the target object is given, and deforms the template model to be aligned with input data. Most template-based methods [LAGP09, ZNI^∗14, GXW^∗15, GXW^∗18] are based on embedded deformation (ED) graph model [SSP07], which is also utilized in templateless works [LVG^∗13, DTF^∗15, DKD^∗16, DDF^∗17]. High-end reconstruction systems using a number of cameras can capture a complete mesh of the target object at each timestamp, so they commonly use template tracking between keyframes for compressing redundant data [CCS^∗15, PKC^∗17, GLD^∗19].

In this paper, we use a template-based method [GXW^∗15, GXW^∗18] to track the motion of a dynamic object, while any method can be used if it can generate a sequence of meshes with a fixed connectivity conforming a RGB-D image sequence.

Given a triangular mesh $\mathcal{M}$ of the template model, we assume that a non-rigid deformation function $\Psi_{t}$ is given for each time frame $t$, satisfying $\Psi_{t}(\mathcal{M})=\mathcal{M}_{t}$, where $\mathcal{M}_{t}$ denotes the deformed template mesh fitted to the depth map $D_{t}$. Our goal is to produce a spatiotemporal (time-varying) texture map $\mathcal{T}$ for the dynamic template mesh. By embedding $\mathcal{M}$ onto a 2D plane using a parameterization method [Mic11], each triangle face $f$ of $\mathcal{M}$ is mapped onto a triangle in the 2D texture domain. A spatiotemporal texture map can be represented as stacked texture maps (or spatiotemporal texture volume), whose size is $W\times W\times T$, where $W$ is the side length of a rectangular texture map and $T$ is the number of input frames. We assume that our mesh $\mathcal{M}_{t}$ keeps a fixed topology in the deformation sequence, and the position of face $f$ in the 2D texture domain remains the same regardless of time $t$. Only the texture information of face $f$ changes with time $t$ to depict dynamically varying object appearances.

To determine the texture information for face $f$ of the deformed mesh $\mathcal{M}_{t}$ at time $t$ in a texture volume, we need to solve two problems: color image selection and texture coordinate determination. We should first determine which color image in the input will be used for extracting the texture information. A viable option is to use the color image $\mathcal{C}_{t}$ at time $t$. However, depending on the camera motion, face $f$ may not be visible in $\mathcal{C}_{t}$, and in that case, texture should be borrowed from another color image $\mathcal{C}_{t}^{\prime}$ where the face is visible. Moreover, when there are multiple such images, the quality of input images should be considered, as some images can provide sharper textures than others.

Once we have selected the color image $\mathcal{C}_{t^{\prime}}$, we need to determine the mapped position of $f$ in $\mathcal{C}_{t^{\prime}}$. Since we already have the deformed mesh $\mathcal{M}_{t^{\prime}}$ at time $t^{\prime}$, the mapping of $f$ onto $\mathcal{C}_{t^{\prime}}$ can be obtained by projecting face $f$ in $\mathcal{M}_{t^{\prime}}$ onto $\mathcal{C}_{t^{\prime}}$. However, due to the errors in camera tracking and non-rigid registration, mappings of $f$ onto different input color images may not be completely aligned. If we use such mappings for texture sampling, the temporal texture volume would contain little jitters in the part corresponding to $f$.

An overall pipeline of our system is shown in Figure 1. Starting from RGB-D images and a deforming model, we determine the aligned texture coordinates using global texture optimization (Section 4). Then, to select color images for each face $f$ at all timestamps, we utilize MRF optimization (Section 5). To reduce redundant calculation, we pre-compute and tabulate the measurements that are required to construct the cost function before MRF optimization. Finally, we build a spatiotemporal texture volume and refine textures by conducting post-processing.

Our cost function consists of data term $E_{d}$ and smoothness term $E_{s}$ as follows,

where $F$ is the set of faces in the template mesh and $\mathcal{N}_{f,t}$ is the set of adjacent nodes of $l_{f,t}$ in the MRF graph.

Textures on a dynamic object may have different dynamism depending on their positions. Some parts can have dynamic appearance changes with object motions, e.g., cloth wrinkles under human motions, while other parts may have almost no changes of textures over time, e.g., naked forearms of the human body. We verified this property with an experiment. After we have obtained optimized texture coordinates for a deforming mesh sequence, we can build an initial incomplete spatiotemporal texture volume by simply copying the visible parts in the input images onto the corresponding texture maps (Figure 3a). Then, we compute the SSIM map between adjacent frames in the texture volume on the overlapping areas (Figure 3b). Figure 3d shows changes in the SSIM values over time for the marked parts in Figure 3c. In the blue and cyan colored regions, SSIM fluctuates along time, meaning that these parts have versatile textures. Such parts need to be guaranteed for dynamism in the final spatiotemporal texture, while smooth texture changes would be fine for other parts.

To reflect this property in the MRF optimization, we modify the smoothness terms on MRF edges. We ignore a temporal MRF edge if the labels of the linked nodes exhibit a lower SSIM value, i.e., less correlation. In our temporal selective weighting scheme, the following new temporal smoothness term replaces $E_{temp}(l_{f,t},l_{f,t^{\prime}})$ in Eq. (7):

where $S(f,\cdot)$ is the average SSIM value of the pixels inside face $f$ computed among consecutive frames. $\chi_{\theta_{\Omega}}(a)$ is a step function whose value is 1 if $a\geq\theta_{\Omega}$, and 0 otherwise. This scheme encourages the dynamism of textures over time when there is less correlation among frames.

In a similar way, we update the spatial smoothness term $E_{spa}(l_{f,t},l_{f^{\prime},t})$ in Eq. (7) as follows:

where $S_{m}(f,k)=\min\limits_{k\in\mathcal{L}_{f}}(S(f,k))$. If there is a label with a small SSIM value in $\mathcal{L}_{f}$, the part has a risk of temporal discontinuity due to the updated temporal smoothness term in Eq. (10). Therefore, in that case, we enforce the spatial smoothness term to impose spatial coherence when temporal discontinuity happens. In Figure 3c, the red box is likely to exhibit temporal changes, e.g., mouth opening. Our spatial selective weighting scheme increases the spatial smoothness term for the red box to secure spatially coherent changes of texture.

Since our MRF graph is large to solve, we divide the problem and solve it using alternating optimization. The idea of fixing a subset and optimizing the others is known as block coordinate descent (BCD). In [CK14, TWW^∗16], the BCD technique is applied to uniform and non-uniform MRF topology. Our MRF graph consists of spatial edges connecting adjacent mesh faces at a single frame and temporal edges connecting corresponding mesh faces at consecutive frames. As a result, our MRF topology is spatially non-uniform but temporally uniform. We subdivide the frames into two sets: even-numbered and odd-numbered. We first conduct optimization for even-numbered frames without temporal smoothness terms. After that, we optimize for odd-numbered frames while fixing the labels of even-numbered frames, where temporal smoothness terms become unary terms. We iterate alternating optimization among even- and odd-numbered frames until convergence. To optimize a single set MRF graph (even or odd), we use an open-source library released by Daniel et al. [TWW^∗16].

After labeling is done, we have a frame label per face at each timestamp. Then we fill the spatiotemporal texture atlas volume by copying texture patches from the labeled frames. When there are drastic brightness changes among different frames, spatial seams may happen on label boundaries (Figure 4a). To handle such cases, we apply gradient processing [PKCH18] to texture slices in the volume, resolving spatial seams (Figure 4c).

In Figure 13a, we compare our results with a texture-mapped model using single images. The single image-based texture has black regions since some parts are invisible at a single frame, while our texture map covers the whole surface of the object. Compared to global texture map [KKPL19], our spatiotemporal texture is more appropriate to convey temporal contents (Figure 13b). In particular, wrinkles on the clothes are expressed better. Additional results are shown in Figures 14 and 15.

For a spatiotemporal texture reconstructed using our method, texture quality of originally visible regions can be measured by comparing with the corresponding regions in the input images, as we did in Section 6.3. However, for regions invisible in the input images, there are no clear ground truths. Even if an additional camera is set up to capture those invisible regions, the captured textures are not necessarily the only solutions, as various textures can be applied to a single shape, e.g., eyes may be open or closed with the same head posture.

To evaluate the quality of reconstructed textures for originally invisible regions without ground truths, we conducted a user study using Amazon Mechanical Turk (AMT). We generated a set of triple images using input images, dynamic model rendered with spatiotemporal texture, and dynamic model rendered with global texture [KKPL19]. We picked the triple images to be as similar as possible to each other, while removing the backgrounds of input images. Figure 9a shows an example. We prepared 40 image triples (120 images) and asked to evaluate whether the given images are realistic or not in two ways: One task is to choose a more realistic image among a pair of images. The other is to score how realistic a given single image is on a scale of 1-10. We set 25 workers whose hit approval rates are over 98% to answer a single question. As a result, we collected total 3,000 pairwise comparison results and 3,000 scoring results.

The preference matrix in Table 4 summarizing the pairwise comparison results shows that the input images are clearly preferred to rendered images using reconstructed textures. This result would be inevitable as geometric reconstruction errors have introduced unnaturalness on top of texture reconstruction artifacts. The errors are mainly caused by the template deformation method, which may not completely reconstruct geometric details. For example, in Figure 9a, the sleeve of T-shirt is tightly attached to the arm in the reconstructed mesh. Still, Table 4 shows our spatiotemporal textures dominate global textures, where only texture reconstruction qualities are compared. The scoring results summarized in Figure 9b show similar conclusions.

As mentioned in Section 2, there are learning-based methods to reconstruct textures of objects. However, there are important differences between our method and those methods. Firstly, our method uses a template mesh and an image sequence from a single camera as the input, whereas the learning-based methods use a single image [SHN^∗19, PTY^∗19] or multi-view images [SHN^∗19]. Although it is unfair, we tested the authors’ implementation of [SHN^∗19] on single images of our dataset (Figure 10). The generated textures on the originally invisible areas are clearly worse than our results. Note that multi-view images that can be used in [SHN^∗19] should be captured by multiple cameras at the same time. Then the setting differs from our case of a single camera, making comparison using multi-view images inappropriate. Secondly, due to memory limitation, the networks of [SHN^∗19, PTY^∗19] cannot fully utilize the textures of our 4K dataset. This would lead to poor quality compared to our results. Thirdly, the learning-based methods target only humans, while our method does not have the limitation, as demonstrated in the top two examples of Figure 14. Finally, our method reconstructs a completed texture volume that can be readily used for generating consistent novel views in a traditional rendering pipeline, but synthesized novel views from learning-based methods are not guaranteed to be consistent.

This work was supported by the Ministry of Science and ICT, Korea, through IITP grant (IITP-2015-0-00174), NRF grant (NRF-2017M3C4A7066317), and Artificial Intelligence Graduate School Program (POSTECH, 2019-0-01906).