Multi-Feature Super-Resolution Network for Cloth Wrinkle Synthesis

A physics-based simulation for realistic fabrics includes velocity updating by physical energies [3, 6], time integration [5], collision detection and collision response [4]. These modules are solved separately and time-consuming. To improve the efficiency of this system, researchers have exploited many algorithms such as implicit time integration [5], adaptive remeshing [19] and iterative optimization [20]. Nevertheless, these algorithms still cost the expensive computation to produce rich wrinkles and are labor-consuming to tune mechanical parameters for desired wrinkling behaviors. Recently data-driven methods have drawn much attention as they offer faster cloth animations than the physics-based methods. Based on precomputed data and data-driven techniques, an HR mesh is either directly synthesized, or super-resolved from a physically simulated LR mesh. In the first stream of work, with precomputed data, researchers have investigated many techniques to accelerate the process for new animations, such as a linear conditional model [11, 21] and a secondary motion graph [22]. Additionally, deep learning-based methods [23, 24, 25] are also used to generate garments on human bodies. In the another line of work, researchers have proposed to combine coarse mesh simulations with learned geometric details from paired mesh databases, to generalize the performance to complicated testing scenes. This stream of methods includes wrinkle synthesis depending on bone clusters [10] or human poses [7] for fitted clothes, and linear upsampling operators [12] or low-dimensional subspace with bases [8, 26] for general case of free-flowing cloth. Inspired by these data-driven methods, we propose a deep learning based approach to synthesize wrinkles on coarse simulated meshes, while our approach is independent with poses or skeletons and not limited with tight garments.

To process 3D models for deep learning, there are various representations [27, 28], e.g., voxels, images, point clouds, meshes. Wang et al.[29] used voxel grids with octree-based convolutional neural networks (CNNs) for 3D shape analysis. Su et al.[30] learned to recognize 3D shapes from multi-view images with 2D-CNNs. Representations based on voxels or multi-view images are extrinsic to the shapes, which are sensitive to isometric deformations, like rotation or translation. Instead of rendered images, recent works [13, 31] use a technique called geometry images [18] encoding features of 3D meshes into the 2D domain for 3D object recognition and generation . With a patch-based approach, this technique is easily coped with deep CNNs and thus suitable for our mesh super-resolution task. Geometry images require parameterization for non-rectangular meshes, and we use a padding scheme to avoid mesh distortion. Recently, some researches [32, 33] directly encode triangle meshes with deformation-based features [34, 35] into latent space with applications to shape embedding and synthesis. These methods focus on the deformation of overall meshes. However, our patch-based algorithm aims at learning local details and is independent of the underlying mesh connectivity.

Feature-based methods aim for proper descriptions of irregular 3D meshes, for synthesizing detailed and realistic objects. Conventional data-driven methods [8] simplify the calculation of wrinkle features, by formulating the strain or stress in an LR mesh. As for deep learning, several algorithms have also investigated robust descriptors for wrinkle deformation. Chen et al.[13] and Oh et al.[14] used 3D coordinates to augment coarse meshes with synthesized wrinkles. Wang et al.[25] learned an autoencoder network for cloth using 3D positions. Instead, Santesteban et al.[9] decomposed the cloth deformation into two displacements, a global fit displacement and the wrinkle displacements. In addition to the position or the displacement, Lähner et al.[15] and Zhang et al.[36] learned high frequency details from normal maps. In our approach, we cascade multiple geometric features as shape descriptors embedded in geometry images, including spatial information of the displacement, the normal and temporal information of the velocity.

In the area of single image super-resolution (SISR), deep learning-based techniques [16, 17, 37, 38, 39] have achieved significant breakthroughs in recent years. Convolutional layers [37] are proposed to be more efficient instead of fully-connected structures in SISR, and later on are extended to deep networks using various upsampling layers, e.g., transposed layers [40] and sub-pixel layers [41]. For deep networks, residual learning [16], dense connections [42], adaptive residual scheme [38], and channel attention module [39] are employed to solve the vanishing gradient problem. Our method likewise uses a deep network with residual dense architecture [17] for its performance and efficiency.

In video super-resolution tasks, how to generate temporal consist results is a vital problem. One way is using consecutive frames as inputs [43] or recurrently using previously predicted outputs [44]. More recently, the recurrent mechanism has influenced the field of cloth animations. For instance, Santesteban et al.[9] used recurrent networks based on gated recurrent units to regress garment wrinkles. Recurrent modules need to predict results sequentially, while our technique processes images individually and parallelly, even in arbitrary order. An alternative solution is to synthesize single output with specialized loss terms to constrain the consistency over time. Loss functions using nearby frames help to alleviate temporal discontinuities in video [45]. In fluid generation, Xie et al.[46] utilized a discriminator loss to preserve temporal coherence. For cloth wrinkle synthesis, Lähner et al.[15] proposed a $L1$ loss between the generated normal map and the ground truth at the previous frame. In our work, the cloth meshes are created by physics-based simulation, and thus ground truth motion is available. A kinematic-based loss constraining the estimated velocity and position enables our network to generate realistic wrinkles while keeping the predictions coherent from frame to frame.

Dual-resolution meshes. Before executing cloth simulation for data preparation, we need to set the initial rest state of LR and HR meshes. We obtain the HR mesh by subdividing the edges of the LR one progressively till the desired resolution. In this work, the number of faces in the HR mesh is 16 times as many as the LR mesh. With the rest state LR/HR meshes, we create two sets of dual-resolution frame data via physics-based simulation. The correspondence between them is maintained during the simulation so that they exhibit similar large-scale folding behaviors but differ in the fine-level wrinkles. More details about the synchronized simulation are given in Subsection 4.2.

Dual-resolution geometry images. We convert the paired meshes to dual-resolution geometry images of 9 channels. The embedded descriptors in our images include the displacement $\bm{d}$, the normal $\bm{n}$ and the velocity $\bm{v}$. Different from the original geometry image paper [18], we encode the displacements instead of positions, since we are only interested in the intrinsic shape of the mesh, not its absolute spatial locations. The displacement is defined as the difference between its position in current frame and that in its starting position. The vertex normal is computed by the area-weighted average normals of the faces adjacent to this vertex. Due to the physics-based simulation with the fixed time step, the velocity is naturally calculated using the positions between two frames (the complete calculation of our feature descriptors is provided in Section 1 of the supplementary material). Since these features are not rotation-invariant, we calculate a rigid motion transformation [47] with rotation $\bm{R}$ and translation $\bm{t}$. Then, we apply ($\bm{R},\bm{t}$) to displacement, and $\bm{R}$ is applied to normal and velocity. To reduce the computation cost, we only compute the rigid motion of LR meshes and apply the same $\bm{(R,t)}$ to the HR meshes. To release the internal covariate shift [48], these features are normalized into a range of [0, 1].

Mesh-to-image conversion. For a mesh in its rest state, we find its bounding box in the 2D material space. Inside the bounding box, we then sample an array of $m\times n$ points uniformly. For each sample point inside the mesh, we find the triangle it is located in, and compute its barycentric coordinate (BC) w.r.t. three triangle vertices. BC is unchanged even though a triangle deforms during simulation. When computing features for sample points, BCs are used as weights for interpolating feature values $\bm{(d,n,v)}$ from triangle vertices. For a mesh whose boundary coincides with the bounding box edge, we do the padding operation along boundaries. Otherwise, for sample points outside the mesh but inside the bounding box, their feature values are filled with the nearest non-zero pixels similar to replicate padding. A long skirt example is given in Fig. 2.

Image-to-mesh conversion. After an HR image is synthesized, values in the displacement channels are used to reconstruct the positions of the detailed mesh, while the original topology of that mesh is retained. Due to the padding operation, every vertex in 2D material space has four nearest non-zero sample points. We reconstruct the displacements of vertices by bilinear interpolation. These computed displacements are added to the positions of subdivided mesh vertices in the rest state to obtain wrinkle-enhanced positions. In the end we apply the inverse of the rigid transformation, computed in the mesh-to-image phase, to new positions. As shown in Fig. 2(d), almost no visual differences can be seen. In our quantitative experiments, the geometric reconstruction error is smaller than 1e-4 meter, measured by the vertex mean square error (VMSE).

The high-quality training dataset is equally important for data-driven approaches. In our case, we need to generate corresponding LR/HR mesh pairs in animation sequences by physics-based simulation. In image super-resolution tasks [16, 37], one way to generate training dataset is down-sampling HR images to obtain their corresponding LR ones. However, down-sampling an HR cloth mesh could cause collisions, even though the HR mesh is collision-free. Therefore, it is preferred that two meshes are simulated individually, with all collisions being properly resolved. However, as mentioned in previous works [8, 12], if there is no constraints between two simulations, they will bifurcate to different behaviors because of accumulated higher frequencies generated by finer meshes and numerical errors. There are several ways to formulate synchronized constraints such as testing functions [12, 49]. And our implementation enforces virtual spring constraints and uses multi-resolution dynamic models to construct synchronized simulation for HR meshes.

Our dual-resolution meshes are well aligned in the initial state, because we only add vertices on the edges without changing the mesh shape. The vertices in an LR mesh, called feature vertices, show up in an HR mesh and are used as constraints for synchronized simulation. We first run coarse cloth simulation and record the positions of all feature vertices at total $N$ frames as ${\bm{p}^{l}_{k}}$, ${k=1,\cdots,N}$, where the superscript $l$ stands for the LR. While simulating an HR mesh at the frame $k$, virtual springs are added to connect pairs $(\bm{p}^{l}_{k},\bm{p}^{h}_{k-1})$ of feature vertices between the LR mesh at the frame $k$ and the HR mesh at the frame $k-1$. To pull $\bm{p}^{h}_{k-1}$ towards $\bm{p}^{l}_{k}$, we define an internal force following Hooke’s law as

where $c$ is a spring stiffness constant that can be adjusted depending on how tight the tracking is desired by the user. A large $c$ results in tight tracking of the feature vertices, but not for other vertices. As a side effect the simulated HR mesh has many annoying “spikes”.

Thus, we add a multi-resolution dynamic model to cooperate with virtual springs. Given an HR mesh at level $H_{0}$ (shown as the solid lines in Fig. 3), we construct an LR triangle mesh at level $H_{1}$ (the dashed triangle in Fig. 3). The mesh in $H_{1}$ connects the feature vertices by retaining the topology of the LR mesh. In finite-element simulations, the constitutive model includes internal cloth forces supporting behaviors such as anisotropic stretch or compression [50] and surface bends [6] with damping [19]. For a triangle in the coarse mesh at level $H_{1}$, the in-plane stretching forces $\bm{f}^{s}_{1}=(\bm{f}^{s}_{11},\bm{f}^{s}_{12},\bm{f}^{s}_{13})$ at three vertices are measured by a corotational finite-element approach [50]. The bending forces for two adjacent triangles are added using a discrete hinge model based on dihedral angles, denoted as $\bm{f}^{b}_{1}=(\bm{f}^{b}_{11},\bm{f}^{b}_{12},\bm{f}^{b}_{13},\bm{f}^{b}_{14})$. The triangles in the fine level $H_{0}$ have the same force patterns $\bm{f}^{s}_{0}$ and $\bm{f}^{b}_{0}$ imposed on all particles (including feature vertices). All stretching and bending forces are added accompanying damping forces. In addition, our two-level dynamic models are independent of the force implementations, and would also work with other triangular finite-element methods. As a result, the feature vertices in multi-resolution dynamic models receive the stretch forces from both $\bm{f}^{s}_{0}$ and $\bm{f}^{s}_{1}$, and the bending forces from both $\bm{f}^{b}_{0}$ and $\bm{f}^{b}_{1}$. The rest vertices are only imposed on the forces at level $H_{1}$. With the two-hierarchy dynamics model, modest virtual spring coefficients can make the HR mesh keep pace with the LR mesh in simulation.

We now introduce our MFSR architecture for the image SR tasks of multiple features. With LR/HR images of the form $(\bm{d},\bm{n},\bm{v})^{l}$ and $(\bm{d},\bm{n},\bm{v})^{h}$, our MFSR learns the mappings of different features by image SR networks. One standard methodology is single task learning, which means learning one task at a time. However it ignores a potentially rich source of information available in other tasks. Another option is multi-task learning, which achieves inductive transfer between tasks, with the goal to leverage additional sources to improve the performance of the target task [51]. Our MFSR is a multi-task architecture, consisting of two components: a single shared network, and three task-specific networks. The shared network is designed based on the SR task, whilst each task-specific network consists of a set of convolutional modules, which link with the shared network. Therefore, the features in the shared network, and the task-specific networks, can be learned jointly to maximise the generalisation of the shared representation across multiple SR tasks, simultaneously maximising the task-specific performance.

Fig. 4 shows a detailed visualisation of our MFSR based on residual dense blocks (RDB) [17]. In the shared network, the image SR model consists of four parts: shallow feature extraction, basic blocks, dense feature fusion, and finally upsampling. We use two convolutional layers to extract shallow features, followed by the RDB [17] as the basic blocks, then dense feature fusion to extract hierarchical features, and lastly one bilinear upsampling layer to upscale the height and width of the LR feature maps by 4 times. Different from general SR tasks, we find that pixel shuffle and deconvolution methods cause apparent checkboard artifacts, thereby we use the bilinear method. For basic blocks in our SR network, we employ RDB instead of residual blocks used in [13]. As shown in Fig. 5(a), a residual block learns a mapping function with reference to its input, and therefore can be used to build deep networks to address the problem of vanishing gradients. However, in the residual block a convolutional layer only has direct connection to its precedent layer, neglecting to make full use of all preceding layers. To exploit all the hierarchical features, we choose RDB (see in Fig. 5(b)) that consists of densely connected layers for global feature combination and local feature fusion with local residual learning. More details about RDB are given in [17]. In each task-specific network, we utilize one convolutional layer to map the extracted local and global features to each upsampled descriptor $\bm{d}^{s},\bm{n}^{s}$, and $\bm{v}^{s}$, respectively.

In order to learn the spatial details and temporal consistency of the underlying HR meshes, our MFSR is trained by minimizing the following loss functions for mesh features. A baseline mean square error (MSE) reconstruction loss is defined as

where $\bm{d}^{h}$ and $\bm{d}^{s}$ stand for the ground truth HR and the synthesized SR displacement images, respectively. This displacement loss term is able to obtain a smooth HR result with given low frequency information.

To extend the loss into wrinkle feature space, a novel ${L}_{2}$ loss for normal is introduced:

where $\bm{n}^{h}$ denotes the ground truth normal image and $\bm{n}^{s}$ denotes the superresolved normal image. The normal feature is directly related to the bending behavior of cloth meshes. This loss term encourages our model to learn the fine-level wrinkle features so that the outputs can stay as close to the ground truth as possible. In our experiments it aids the networks in creating realistic details.

The above two loss terms are utilized to reconstruct high-frequency details exclusively from spatial statistics. To improve the consistency for animation sequences, we should also take the temporal coherence into account. The vertex velocities of every animation frame contribute a velocity loss of the form

where $\bm{v}^{h}$ denotes the ground truth velocity image and $\bm{v}^{s}$ denotes the synthesized velocity image.

In addition, we minimize a kinematics-based loss in the training stage, to constrain the relationship between synthesized velocities and displacements (please refer to the supplementary material for the detail derivation) as

where $n$ is the length of frames associated to the input frame, and $\Delta t$ represents the time step between consecutive frames. $\bm{R}$ is the precomputed rotation part in the rigid motion transformation for input frame. This kinematics-inspired loss term can improve the consistency between the generated cloth animations.

The overall loss of our MFSR is defined as

which is a linear combination of spatial smoothness, detail similarity, temporal consistency and kinematic loss terms with the weight factors $w_{d},w_{n},w_{v}$, and $w_{kine}$. As for back propagation, each loss term propagates backwards through the task-specific layer independently. In the shared layers, parameters are updated according to the total loss $\mathcal{L}_{all}$. As a result, the gradient of loss functions from multiple SR tasks will pass through the shared layers directly, and learn a common representation for all related tasks.

The reconstructed meshes converted from super-resolved images by the above network may suffer from penetrations with obstacles or self-collision. For interactions with human or balls, we adopt a fast refinement method [25] to push the cloth vertices colliding with the obstacles outside and meanwhile preserving the local wrinkle details. As for self-collision, since the runtime simulation of an LR mesh is collision-free, we interpolate the vertices in the penetrating area between LR meshes and the reconstructed ones to guarantee to resolve all collisions. In implementation, we use the bisection method [52] to search for a close-to-optimal interpolation weight. We do the bisection several times and then take the last collision-free state as the interpolation result (see Fig. 6). It is not necessary to let all vertices of the whole mesh get involved in the position interpolation. Instead, only the vertices involved in the intersections are of our interests. These vertices can be specified by a discrete collision detection process and grouped into impact zones as done in [53]. Position interpolations are performed per zone, and each zone has different interpolation weights. In this way, the synthesized meshes are least affected by the collision handling.

We construct three datasets using a tablecloth model and a skirt model with character motions. The two models are regular and irregular garment shapes, respectively. The meshes in each dataset are simulated from a fixed template model. For the tablecloths, we generate two datasets, called DRAPING and HITTING (see Fig. 7). The DRAPING dataset is created by randomly handling one of the topmost vertices of the tablecloth and letting the fabric fall freely. It contains 13 simulation sequences, each with 400 frames. 10 sequences are randomly selected for training and remaining 3 sequences are for testing. In addition to simulating a piece of tablecloth in a free environment, we also construct a HITTING dataset where a sphere interacts with the tablecloth. Specifically, we select spheres of different sizes to hit the tablecloth back and forth at different locations, and obtain a total of 35 simulation sequences, with 1,000 frames for each sequence. We randomly select 27 sequences for training and 8 sequences for testing. The SKIRT dataset is created by the long skirt garments worn by an animated character (shown in Fig. 7). A mannequins has rigid parts as [19] and is driven by publicly available motion capture data from CMU ¹¹1http://mocap.cs.cmu.edu. We select dancing motions including 7 sequences (in total 30,000 frames), in which 5 sequences are randomly selected for training and 2 sequences are for testing. To simulate cloth stably, we interpolate 8 times between two adjacent motions from the original data.

We apply the ARCSim engine [19] to produce all simulations with remeshing disabled. A material called the Gray Interlock is adopted for its anisotropic behaviors, from a library of measured cloth materials [54]. To meet a collision-free initial state for skirts, we first manually put the skirt on a template mannequin (T pose), and then interpolate 80 frames between the T pose and the initial poses of motion sequences. In addition, for synchronized simulation, we set the spring stiffness constant $c=10$ in the equation (1).

For different datasets, we train each model separately. Our proposed MFSR consists of shared and task-specific layers. The shared network has 16 identical RDB [17], where six of them are densely connected layers for each RDB, and the growth rate is set to 32. The basic network settings, such as the convolutional kernel and the activation function, are set according to [17]. For the upscaling operation, i.e., from the coarse resolution features to fine ones, we consider several different mechanisms, e.g., pixel shuffle module [41], deconvolution, nearest and bilinear, and finally choose the bilinear upscaling layer because it can prevent checkerboard artifacts in the generated meshes. In our upsampling network, the upscale factor is set to 4. The upscale factor (in one dimension) for corresponding meshes is set to be as close to 4 as possible. For example, the LR and the HR tablecloth meshes have 749 and 11,393 vertices, respectively, and the latter is roughly 16 times as many as the former. Converting meshes to images, we set the size of LR images in tablecloth to be $192\times 128$, and HR ones $768\times 512$. The image aspect ratio is the same to the UV proportion in material space to achieve uniform sampling.

We implement our network using PyTorch 1.0.0. In each training batch, we randomly extract 16 LR/HR pairs with the size of $72\times 72$ and $288\times 288$ as input. Adam optimizer [55] is used to train our network, and its $\beta_{1}$ and $\beta_{2}$ are both set to 0.9. The base learning rate is initialized to 1e-4, and is divided by 10 every 20 epochs. To avoid the learning rate becoming too small, we fix it after 60 epochs. The training procedure stops after 120 epochs and takes about a day and a half. In all our experiments, we set the length of the input frames $n=3$ for the kinematics-based loss in the equation (6). Besides, we set the weights $w_{d}=0.9,w_{n}=0.03,w_{v}=0.03$ and $w_{kine}=0.03$ in the equation (6).

We implement our method on a 2.50GHz Core 4 Intel CPU for coarse simulation and mesh-image conversion, and a NVIDIA GeForce^® GTX 1080Ti GPU for image synthesizing. Table 1 shows average per-frame execution time of our method for different garment resolutions. The execution time contains four parts: coarse simulation, mesh/image conversion, image synthesizing, and refinement. For reference, we also calculate the simulation timings of a CPU-based implementation of tracked high-resolution simulation using ARCSim [19]. Our algorithm is averagely 13 times faster than the tracked simulation. The low computational cost of our method makes it suitable for the interactive applications.

Generalization to new hanging. We use the training data in the DRAPING dataset to learn a synthesizer, and then evaluate the generalization to new hanging vertices. Fig. 8 shows the deformations of tablecloths of three test sequences in the DRAPING dataset. We compare our results with the HR meshes of tracked physics-based simulation. Our approach successfully produces the realistic and abundant wrinkles in different deformation sequences. For instance, tablecloths appear middle and small wrinkles when falling from different directions.

Generalization to new balls. Fig. 9 presents the performance of our algorithm in the HITTING dataset, which illustrates the performance when generalizing to new crashing balls of various sizes and initial positions. We show four test examples comparing the ground-truth HR of the tracked simulation with our method. For testing, the initial positions of balls are set to four different places which are unseen in training data. Additionally, in the last two columns of Fig. 9, the diameter of the ball is set to 0.5 which is a new size not used for training. When various sizes of balls crash into the cloth in different positions, our method can successfully predict the plausible wrinkles, with 12 times faster running speed than physics-based simulation.

Generalization to new motions. In Fig. 10, we show the deformed long skirt produced by our approach on the mannequins while changing various poses over time. The human poses are from two testing motion sequences $05\_04$ in the subject of modern dance and $55\_02$ in the subject of lambada dance ²²2http://mocap.cs.cmu.edu. We visually compare the results of our algorithm with the ground-truth simulation. The mid-scale wrinkles are successfully predicted by our approach when generalizing to various dancing motions not in the training set. For instance, in the first column of Fig. 10, the skirt slides forward and forms plausible wrinkles due to an extended and straight leg caused by the character pose of sideways arabesque. As for dancing sequences, please see the accompanying video for more animated results and further comparisons.

Comparison with other methods. Given detailed meshes simulated by the physics-based technique as ground truth, we compare our results with our implementation of a CNN-based method [13] and a conventional machine learning-based method [8]. The performance is evaluated on the Tablecloth dataset combining DRAPING and HITTING by a single network. The partition of the dataset for training and testing and the training parameters of our method are the same with the setting illustrated in Subsection 6.2.

We train the network of Chen et al.[13] with the same setting reported in their paper. The peak signal-to-noise ratio (PSNR) and the vertex-wise mean square error (VMSE) are used to evaluate the quality of reconstructions, quantitatively. As shown in Table 2, our MFSR gains better performance than [13] with a higher PSNR and a lower VMSE. We also show comparison results in Fig. 11, with color coding to highlight the distance between the predicted results and the ground truth. Feeding the coarse meshes, our MFSR successfully produces rich and consistent wrinkles thanks to displacement, normal and velocity features with corresponding loss functions. Chen et al.[13] are able to predict the overall wrinkles of the draping and hitting tablecloth. However, there are unsmooth triangles leading to spatial and temporal artifacts since no temporal loss modules are applied. This leads to unstable animations, please refer to the accompanying video. The difference map also clearly indicates that the results of Chen et al.[13] highlight the bottom left and right corners and wrinkle lines, where our method leads to significantly lower reconstruction errors. Our MFSR model provides better visual quality for wrinkle synthesis and the generated HR results look closer to the ground truth.

We further make comparison with state-of-the-art conventional machine learning-based method [8] (not deep learning-based) for cloth wrinkle synthesis. Since images are not utilized by Zurdo et al.[8], we use a metric, vertex-wise mean square error (VMSE), for quantitative comparison. As shown in Table 2, our results gain lower VMSE than the results of Zurdo et al.[8]. The results of [8] show inaccurate and unexpected rough wrinkles different from the ground truth, e.g., the left and right sides in the first row of Fig. 11 (c). Their conventional example-based algorithm is reliant on a very limited number of examples and parameters. Thus their method can reconstruct good results while the testing samples are near the example poses, but may cause unexpected artifacts for more diverse testing data. The accompanying video show that the results of [8] have small artifacts and are lack of temporal coherence. This is due to their designed mesh descriptor, the edge ratio between current and the rest state, which only encodes quasistatic wrinkle formation without dynamic features. Our method does not have such drawbacks with the help of spatial and dynamic descriptors as inputs and temporal loss constraints. Notice how our method successfully reconstructs the stable and realistic cloth animations. Besides, collisions are not handled in their work. Our approach solves cloth penetrations with controllable cost (see in Table 1).

We also evaluate our method on irregular meshes. We take pants as an example, since they are not topological a disk and thus cannot be parametrized into one plane. Fig. 12 and Fig. 13 show our method for pants. We follow the general idea of geometry images [18] to cut and open the mesh into patches. The process of unfolding 3D cloth models into 2D patches is especially natural since 3D garments usually have corresponding 2D patterns in garment design. Fig. 12 (a) shows the four 2D patterns consisted of the pants, and Fig. 12 (b) shows the reconstructed mesh from 12 (a) with clear artifacts at the seam lines. Our proposed padding algorithm in Subsection 4.1 fills the zero pixels for learning and reconstruction. In Fig. 12 (c), we can see the well reconstructed mesh from padded geometry images. Fig. 13 shows a frame result of synthesized pants with our method, compared with input coarse mesh and the ground truth. It can be seen that our method reconstructs middle-scale wrinkles.

Next, we study the effect of different components of our proposed network, including loss function and network architecture.

Loss function. To demonstrate the effectiveness of our proposed loss functions, we conduct the experiments with different loss combinations on three datasets, i.e., DRAPING, HITTING, and SKIRT, respectively. The training and testing datasets are selected as mentioned in Subsection 6.1. We use the displacement loss as the baseline and progressively add the remaining loss terms of our MFSR, to obtain the comparative results.

Table 3 reports quantitative evaluations of our proposed loss functions. In several settings of loss functions, we compare the PSNR between generated displacement images and the ground truth, and the VMSE between synthesized meshes and the ground truth. Red text indicates the best performance and the blue text indicates the second-best. The result shows that our algorithm has either the best or second-best performance through combining all loss terms in a multi-task learning framework. Notice that without the constraints of velocity and kinematics-based loss, the results with $\mathcal{L}_{n}$ gain lower PSNR and higher VMSE although it encourages wrinkle generation in SR results.

In Fig. 14, we show visual results from the DRAPING dataset, to evaluate the performance of different loss combinations. The baseline model (see Fig.14(a)) only including displacement loss generates cloth meshes with too many unrealistic small wrinkles and inconsistent thin lines. Directly combining velocity loss with the baseline (see Fig.14(c)) is not able to solve this problem, contrarily introducing some unexpected horizontal short lines. The results including normal loss also suffer from uneven lines and sharp triangles on the folders and boundaries (i.e., several buckling triangles near the handler and center wrinkle lines on the top of Fig.14(b) and 14(d)). Finally, the results with all our proposed loss functions (see Fig.14(e)) are visually very close to the ground truth (see Fig.14(f)) with the realistic and consistent wrinkles and folders, which verifies the effectiveness of our proposed four loss functions.

Network architecture. To further investigate the performance of different networks (SRResNet and RDN), we conduct experiments on the DRAPING dataset. In particular, we validate our cloth animation results on randomly selected 800 pairs of LR/HR meshes from the DRAPING dataset, which are excluded from the training set, and cover different complex hanging motions in pendulum movement. In Fig. 15, we depict the convergence curves of three different features in the above validation dataset. The convergence curves show that RDN achieves better performance than SRResNet, and further stabilizes the training process in all three features. The improved performance and stabilization are benefited from contiguous memory, local residual learning and global feature fusion in RDN. In SRResNet, local convolutional layers do not have direct access to the subsequent layers, thereby it neglects to fully use the information of each convolutional layer. As a result, RDN achieves better performance than SRResNet.