NRST: Non-rigid Surface Tracking from Monocular Video

Given the template $\hat{\mathbf{V}}$ and our estimate of the previous frame $\mathbf{V}^{t}$, our method sequentially estimates the geometry $\mathbf{V}^{t+1}$ of the current frame $t+1$. We jointly optimize per-vertex local rotations denoted by $\mathbf{\Phi}^{t+1}$ and vertex locations $\mathbf{V}^{t+1}$. Specifically, for each time step the deformation estimation is formulated as the non-linear optimization problem

$$(\mathbf{V}^{t+1},\mathbf{\Phi}^{t+1})=\operatorname*{arg\,min}_{\mathbf{V},\mathbf{\Phi}\in\mathbb{R}^{N\times 3}}E\left(\mathbf{V},\mathbf{\Phi}\right)\text{,}$$

(1)

with

$\displaystyle\begin{split}E\left(\mathbf{V},\mathbf{\Phi}\right)&=\lambda_{\mathrm{Photo}}E_{\mathrm{Photo}}\left(\mathbf{V}\right)+\lambda_{\mathrm{Smooth}}E_{\mathrm{Smooth}}\left(\mathbf{V}\right)+\lambda_{\mathrm{Edge}}E_{\mathrm{Edge}}\left(\mathbf{V}\right)\\ &+\lambda_{\mathrm{Arap}}E_{\mathrm{Arap}}\left(\mathbf{V},\mathbf{\Phi}\right)+\lambda_{\mathrm{Vel}}E_{\mathrm{Vel}}\left(\mathbf{V}\right)+\lambda_{\mathrm{Acc}}E_{\mathrm{Acc}}\left(\mathbf{V}\right)\text{.}\end{split}$

(2)

$\lambda_{Photo}$, $\lambda_{Smooth}$, $\lambda_{Edge}$, $\lambda_{Vel}$, $\lambda_{Acc}$, $\lambda_{Arap}$ are hyperparameters set before the optimization starts and afterwards they are kept constant. $E\left(\mathbf{V},\mathbf{\Phi}\right)$ combines different cost terms ensuring that the mesh deformations agree with the motion in the video. The resulting non-linear least squares optimization problem is solved with the GPU-based Gauss-Newton solver based on the work of Zollhoefer et al. [44] where we adapted the Jacobian and residual implementation to our energy formulation. The high efficiency is obtained by exploiting the sparse structure of the system of normal equations. For more details we refer the reader to the approach of Zollhoefer et al. [44]. Now, we will explain the terms in more detail.

Photometric Alignment. The photometric term

$\displaystyle\begin{aligned} E_{\mathrm{Photo}}(\mathbf{V})=\sum_{i=1}^{N}\left\lVert\sigma\left(I^{t+1}\ast\mathbf{G}_{w}\left(\Pi\left(\mathbf{V}_{i}\right)\right)-\mathbf{C}_{i}\right)\right\rVert&{}^{2}\end{aligned}$

(3)

densely measures the re-projection error. $\left\lVert\cdot\right\rVert$ is the Euclidean norm, $\ast$ is the convolution operator and $\mathbf{G}_{w}$ is a Gaussian kernel with standard deviation $w$. We use Gaussian smoothing on the input frame for more stable and longer range gradients. $\Pi(\mathbf{V}_{i})=(\frac{u}{w},\frac{v}{w})^{\top}$ with $(u,v,w)^{\top}=\mathbf{I}\mathbf{V}_{i}$ projects the vertex $\mathbf{V}_{i}$ on the image plane and $I^{t+1}\ast\mathbf{G}_{w}$ returns the RGB color vector of the smoothed frame at position $\Pi\left(\mathbf{V}_{i}\right)$ which is compared against the pre-computed and constant vertex color $\mathbf{C}_{i}$. Here, $\mathbf{I}\in\mathbb{R}^{3\times 3}$ is the intrinsic camera matrix. $\sigma$ is a robust pruning function for wrong correspondences with respect to color similarity. More specifically, we discard errors above a certain threshold because in most cases they are due to occlusions.

Spatial Smoothness. Without regularization, estimating 3D geometry from a single image is an ill-posed problem. Therefore, we introduce several spatial and temporal regularizers to make the problem well-posed and to propagate 3D deformations into areas where information for data terms is missing, e.g., poorly textured or occluded regions. The first prior

$$E_{\mathrm{Smooth}}\left(\mathbf{V}\right)=\sum_{i=1}^{N}\sum_{j\in\mathcal{N}(i)}\left\lVert\left(\mathbf{V}_{i}-\mathbf{V}_{j}\right)-(\hat{\mathbf{V}}_{i}-\hat{\mathbf{V}}_{j})\right\rVert^{2}$$

(4)

ensures that if a vertex $\mathbf{V}_{i}$ changes its position, its neighbors $\mathbf{V}_{j}$ with $j\in\mathcal{N}(i)$ are deformed such that the overall shape is still spatially smooth compared to the template mesh $\hat{\mathbf{V}}$. In addition, the prior

$$E_{\mathrm{Edge}}\left(\mathbf{V}\right)=\sum_{i=1}^{N}\sum_{j\in\mathcal{N}(i)}\left(\left\lVert\mathbf{V}_{i}-\mathbf{V}_{j}\right\rVert-\lVert\hat{\mathbf{V}}_{i}-\hat{\mathbf{V}}_{j}\rVert\right)^{2}$$

(5)

ensures isometric deformations which means that the edge length with respect to the template is preserved. In contrast to $E_{\mathrm{Smooth}}$, this prior is rotation invariant. Finally, the as-rigid-as-possible (ARAP) prior [34]

$$E_{\mathrm{Arap}}\left(\mathbf{V},\mathbf{\Phi}\right)=\sum_{i=1}^{N}\sum_{j\in\mathcal{N}(i)}\left\lVert\left(\mathbf{V}_{i}-\mathbf{V}_{j}\right)-\mathcal{R}(\mathbf{\Phi}_{i})(\hat{\mathbf{V}}_{i}-\hat{\mathbf{V}}_{j})\right\rVert^{2}\text{,}$$

(6)

allows local rotations for each of the mesh vertices as long as the relative position with respect to their neighborhood remains the same. Each row of the matrix $\mathbf{\Phi}\in\mathbb{R}^{N\times 3}$ contains the per-vertex Euler angles which encode a local rotation around $\mathbf{V}_{i}$. $\mathcal{R}(\mathbf{\Phi}_{i})$ converts them into a rotation matrix.

We choose a combination of spatial regularizers to ensure that our method can track different types of non-rigid deformations equally well. For example, $E_{\mathrm{Smooth}}$ is usually sufficient to track facial expressions without large head rotations. But tracking rotating objects can only be achieved with rotational invariant regularizers ($E_{\mathrm{Edge}}$, $E_{\mathrm{Arap}}$). In contrast to Yu et al. [43], we adopt the Euclidean norm in Eq. 4 and Eq. 6 instead of the Huber loss because it led to visually better results.

Temporal Smoothness. To enforce temporally smooth reconstructions, we propose two additional priors. The first one is defined as

$$E_{\mathrm{Velocity}}\left(\mathbf{V}\right)=\sum_{i=1}^{N}\left\lVert\mathbf{V}_{i}-\mathbf{V}_{i}^{t}\right\rVert^{2}$$

(7)

and ensures that the displacement of vertices between $t$ and $t+1$ is small. Second, the prior

$$E_{\mathrm{Acc}}\left(\mathbf{V}\right)=\sum_{i=1}^{N}\left\lVert\left(\mathbf{V}_{i}-\mathbf{V}_{i}^{t}\right)-\left(\mathbf{V}_{i}^{t}-\mathbf{V}_{i}^{t-1}\right)\right\rVert^{2}$$

(8)

penalizes large deviations of the current velocity direction from the previous one.

Tracking of uniformly colored fabrics is usually challenging for classical color-based terms due to the lack of color features. To overcome this limitation, we inspected the structure of different garments and found that most of them show line-like micro-structures due to the manufacturing process of the woven threads, see Fig. 2 left. Those can be recorded with recent high resolution cameras such that reconstruction algorithms can make use of those patterns. To this end, we propose a novel texture term to refine the estimation of non-rigid motions for the case of woven fabrics. It can be combined with the terms in Eq. 2. Now, we will explain our novel data term in more detail.

Histogram of Oriented Gradient (HOG). Based on HOG [5] we compute for each pixel $(i,j)$ of an image the corresponding histogram $\mathbf{h}_{i,j}\in\mathbb{R}^{b}$ where $b$ is the number of bins that count the total number of gradient angles present in the neighborhood of pixel $(i,j)$. To be more robust with respect to outliers and noise we count the number of gradients per angular bin irrespective of the gradient magnitude and only if the magnitude is higher than a certain threshold. Compared to pure image gradients, HOG is less sensitive to noise. Especially for woven fabrics, image gradients are very localized since changing the position in the image can lead to large differences in the gradient directions due to the high frequency of the image content. HOG instead averages over a certain window so that outliers are discarded.

Directions of a Texture. Applying HOG to pictures of fabrics reveals their special characteristics caused by the line like patterns (see Fig. 2). There are two dominant texture gradient angles $\alpha$ and $\beta=((\alpha+180)\mod 360)$ perpendicular to the lines. So $\alpha$ provides the most characteristic information of the pattern in the image at $(i,j)$ and can be computed as the angle whose bin has the highest frequency in $\mathbf{h}_{i,j}$. $\alpha$ is then converted to its normalized 2D direction, also called dominant frame gradient (DFG), which is stored in the two-valued image $I_{\mathrm{Dir}}(i,j)$. To detect image regions that do not contain line patterns, we set $I_{\mathrm{Dir}}(i,j)=(0,0)^{\top}$ if the highest frequency is below a certain threshold.

Texture-based Constraint. Our novel texture term

$$\begin{split}E_{\mathrm{Tex}}\left(\mathbf{V}\right)=\sum_{i=1}^{F}\left\lVert\rho\left(\mathbf{d}_{\mathrm{M},i},\mathbf{d}_{\mathrm{F},i}\right)\right\rVert^{2}\end{split}$$

(9)

ensures now that for all triangles the projected DFG $\mathbf{d}_{\mathrm{M},i}$ parametrized on the object surface agrees with the frame’s DFG $\mathbf{d}_{\mathrm{F},i}$ at the location of the projected triangle center. An overview is shown in Fig. 3. More precisely, by averaging $\mathbf{U}_{k},\mathbf{U}_{m},\mathbf{U}_{l}$ one can compute the pixel position $\mathbf{z}_{\mathrm{TM},i}\in\mathbb{R}^{2}$ of the center point of the triangle $\mathbf{F}_{i}=(k,m,l)$ in the texture map. Now, the neighborhood region for HOG around $\mathbf{z}_{\mathrm{TM},i}$ is defined as the 2D bounding box of the triangle. The HOG descriptor for $\mathbf{z}_{\mathrm{TM},i}$ can be computed and by applying the concept explained in the previous paragraph one obtains the DFG $\mathbf{d}_{\mathrm{TM},i}$ (see Fig 3 (a)). Next, we define $\mathbf{b}_{\mathrm{TM},i}=\mathbf{z}_{\mathrm{TM},i}+\mathbf{d}_{\mathrm{TM},i}$ and express it as a linear combination of the triangles’ UV coordinates leading to the barycentric coordinates $\mathbf{B}_{i,1},\mathbf{B}_{i,2},\mathbf{B}_{i,3}$ of the face $\mathbf{F}_{i}$. They form together with the other triangles the barycentric coordinates matrix $\mathbf{B}\in\mathbb{R}^{F\times 3}$. Each row represents the texture map’s DFG for the respective triangle of the mesh in an implicit form. Since $\mathbf{b}_{\mathrm{TM},i}$ can be represented as a linear combination, one can compute the corresponding 3D point $\mathbf{b}_{\mathrm{3D},i}=\mathbf{B}_{i,1}\mathbf{V}_{k}+\mathbf{B}_{i,2}\mathbf{V}_{m}+\mathbf{B}_{i,3}\mathbf{V}_{l}$ as well as the triangle center $\mathbf{z}_{\mathrm{3D},i}\in\mathbb{R}^{3}$ in 3D (see Fig 3 (b)). The barycentric coordinates remain constant, so that $\mathbf{b}_{\mathrm{3D},i}$ and $\mathbf{z}_{\mathrm{3D},i}$ only depend on the mesh vertices $\mathbf{V}_{k}$, $\mathbf{V}_{m}$ and $\mathbf{V}_{l}$. One can then project the DFG of the mesh $\mathbf{d}_{\mathrm{M},i}=\Pi(\mathbf{b}_{\mathrm{3D},i})-\Pi(\mathbf{z}_{\mathrm{3D},i})$ into the frame and compare it against the DFG $\mathbf{d}_{\mathrm{F},i}$ of the frame $t+1$ at the location of $\Pi(\mathbf{z}_{\mathrm{3D},i})$ which can be retrieved by an image lookup in $I_{\mathrm{Dir}}^{t+1}$ (see Fig 3 (c)). $\rho\left(\mathbf{x},\mathbf{y}\right)$ computes the minimum of the differences between $\mathbf{x},\mathbf{y}$ and $\mathbf{x},-\mathbf{y}$ iff both $\mathbf{x}$ and $\mathbf{y}$ are non-zero vectors (otherwise we are not in an area with line patterns) and the directions are similar up to a certain threshold to be more robust with respect to occlusions and noise. As mentioned above, there are two DFGs in the frame for the case of line patterns. We assume the initialization is close to the ground truth and choose the minimum of the two possible directions.

Now, we evaluate NRST on datasets for general objects like faces where we disable $E_{\mathrm{Tex}}$. After that, we compare our approach against another monocular method. Finally, we evaluate our proposed texture term on two new scenes showing line-like fabric structures, perform an ablation study and demonstrate interesting applications. More results can be found in the supplemental video.

Qualitative Evaluation for General Objects. In Fig. 4 we show frames from our monocular reconstruction results. We tested our approach on two face sequences [43, 39] where templates are provided. Note that NRST precisely reconstructs facial expressions. The 2D overlay (second column) matches the input and also in 3D (third column) our results look realistic. Furthermore, we evaluated on the datasets of Varol et al. [40] and Salzmann et al. [31] showing fast movements of a T-shirt and a waving towel. Again for most parts of the surface the reconstructions look accurate in 2D since they overlap well with the input and they are also plausible in 3D. This validates that our approach can deal with the challenging problem of estimating 3D deformations from a monocular video for general kinds of objects.

Comparison to Yu et al. [43]. Fig. 5 shows a qualitative comparison between our method and the one of Yu et al. [43]. It becomes obvious that both capture the facial expression, but the proposed approach is faster than the one of Yu et al. due to our data-parallel GPU implementation. In particular, on their sequence our method runs at 15fps whereas their approach takes several seconds per frame. More sidy-by-side comparisons on this sequence can be found in the supplemental video.

Qualitative Evaluation for Fabrics. The top row of Fig. 6 shows frames (resolution $1920\times 1080$) of a moving piece of cloth that has the typical line patterns. Although the object is sparsely textured, our approach is able to recover the deformations due to the texture term, which accurately tracks the DFG of the line pattern. As demonstrated in the last column, the estimated angles for the frames are correct and therefore give a reliable information cue exploited by $E_{\mathrm{Tex}}$. For quantitative evaluation, we created a synthetic scene that is modeled and animated in a modeling software showing a carpet that has the characteristic line pattern but is also partially textured (see bottom row of Fig. 6). We rendered the scene at a resolution of $1500\times 1500$. $E_{\mathrm{Tex}}$ helps in the less textured regions where $E_{\mathrm{Photo}}$ would fail. The last column shows how close our reconstruction (red) is with respect to ground truth (blue).

Ablation Analysis. Apart from the proposed texture term, our energy formulation is similar to the one of Yu et al. [43]. To validate that $E_{\mathrm{Tex}}$ improves the reconstruction over a photometric-only formulation, we perform an ablation study. We measured the averaged per-vertex Euclidean distance between the ground truth mesh and our reconstructions. For the waving towel shown in Fig. 6 bottom, we obtained an error of 26.8mm without $E_{\mathrm{Tex}}$ and 25.5mm if we also use our proposed texture term leading to an improvement of 4.8%. The diagonal of the 3D bounding box of the towel is 3162mm. For the rotation sequence (resolution $800\times 800$) shown in Fig. 7 the color variation is very limited since background and object have the same color. In contrast to $E_{\mathrm{Photo}}$ alone, $E_{\mathrm{Tex}}$ can rotate the object leading to an error of 4.1mm for the texture-only case and 6.7mm for the photometric-only setting. So $E_{\mathrm{Tex}}$ improves over $E_{\mathrm{Photo}}$ by 38.8%

Our method enables several applications such as free view point rendering or re-texturing on general deformable objects or for virtual face make-up (see Fig. 8). Since our approach estimates the deforming geometry, one can even change the scene lighting for the foreground such that the shading remains realistic.

By the nature of the challenging task of monocular tracking of non-rigid deformations, our method has some limitations which open up directions for future work. Although, our proposed texture term uses more of the information contained in the video than a photometric-only formulation, there are still image cues that can improve the reconstruction like shading and the object contour as demonstrated by previous work [13, 41]. So, one could combine them in a unified framework. To increase robustness, the deformations could be jointly estimated over a temporal sliding window as proposed by Xu et al. [41] and an embedded graph [35] could lead to improved stability by reducing the number of unknowns.