Liquid Warping GAN with Attention: A Unified Framework for Human Image Synthesis

We summarize the recent image-to-image translation-based and the warping-based methods as follows.

Image-to-Image translation-based methods. Esser et al.[26] use a Variational U-Net to learn a mapping function from a 2D skeleton to an image. Chan et al.[9] learn a mapping function from a 2D skeleton to an image by a pix2pixHD [27] with a specialized Face GAN and temporally coherent GAN. Wang et al.[28] propose a vid2vid framework and learn a mapping function from 2D dense pose to image. Meanwhile, Shysheya et al.[29] firstly build a full texture UV image of a person by multi-view cameras, then learn a mapping function from a 3D skeleton to part coordinates of the UV map and finally render a result based on the coordinates and the UV image. Contemporarily, Liu et al.[30] firstly use a monocular video to reconstruct a full 3D character model of a person with a static pose, then render the texture of each body parts and finally learn a mapping from synthetic to real images. However, all these methods train a mapping from keypoints or parts to each person’s image and everybody needs to train their own model. This might limit its wide application.

Warping-based methods. Recent work is mainly based on the conditioned generative adversarial networks (CGAN) [2, 1, 31, 14, 32]. Their key technical idea is to combine the source image along with the source pose (2D skeleton) as inputs and generate a realistic image by GANs using a reference pose. The differences among those approaches are merely in network architectures, warping strategies, and adversarial losses. In [2], Ma et al.[2] directly concatenate the source image and the reference pose, and then design a U-Net [33] generator with a coarse-to-fine strategy to generate $256\times 256$ images. Neverova et al. [14] replace the sparse 2D key points with the dense correspondences between the image and surface of the human body by the DensePose [34]. Si et al. [32] propose a multistage adversarial loss and separately generate the foreground (or different body parts) and background. Balakrishnan et al.[1] firstly fit an affine transformation matrix based on the source and the target 2D key points and then use a texture warping strategy to generate the foreground and the background separately. These work [3, 11, 35, 13], focus on the way of warping the source features into the target conditions, like skeleton or parsing. Besides, Li et al.[36] propose to learn a transformation flow from 2D key points and warp the deep features based on the learned transformations.

Human appearance modeling or transfer is a vast topic, especially in the field of virtual try-on applications, from computer graphics pipelines [37] to learning based pipelines [4, 5]. Graphics based methods first estimate the detailed 3D human mesh with clothes via garments and 3D scanners [38] or multiple camera arrays [39], and then human appearance with clothes is capable of being conducted from one person to another based on the detailed 3D mesh. Although these methods can produce high-fidelity results, their cost, size, and controlled environment are unfriendly and inconvenient to customers. Recently, in the light of deep generative models, SwapNet [4] firstly learns a pose-guided clothing segmentation synthetic network, and then the clothing parsing results with texture features from the source image are fed into an encoder-decoder network to generate the image with the desired garment. In [5], the authors leverage a geometric 3D shape model combined with learning methods, swap the color of visible vertices of the triangulated mesh, and train a model to infer that of invisible vertices. Instead of estimating the 3D clothes by other sensors, in the MGN [40], the authors, train a network with 3D scans data and predict the body shape and clothing directly from 8 frames or a video. They apply the garment transfer based on the estimated 3D body mesh with clothes.

Novel view synthesis aims to synthesize new images of the same object or human body from arbitrary viewpoints. The core step of existing methods is to fit a correspondence map from the observable views to new views with convolutional neural networks. In [41], the authors use CNNs to predict appearance flow and synthesize new images of the same object by copying the pixel from a source image based on the appearance flow and they have achieved decent results of rigid objects like vehicles. The following work [42] proposes to infer the invisible textures based on appearance flow and adversarial generative network (GAN) [43], while Zhu et al. [7] argue that appearance flow-based method performs poorly on articulated and deformable objects, such as human bodies. They propose an appearance-shape-flow strategy to synthesize different views of human bodies – besides, Zhao et al. [6] design a GAN based method to synthesize high-resolution views in a coarse-to-fine way. Recently, in PiFu [44], the authors learn an implicit function with multi-layer perceptrons (MLPs) to digitize the human body and infer the 3D surfaces and texture from a single or multiple frames. The fully digitalized human body could synthesize a different view.

Ding et al. [45] propose a generative adversarial one-shot face recognizer to synthesize new face images. Shaham et al. [24] introduce a SinGAN, an unconditional generative model from a single image. Zakharov et al. [25] apply the few-shot adversarial learning to generate the realistic talking head. In light of the success of the Meta-Learning in classification, reinforcement learning and network architecture search [46, 47, 48], Lee et al. [10] propose a MetaPix for the few-shot motion imitation. Wang et al. [49] extend the previous vid2vid [28] framework within a few-shot setting and make it capable of synthesizing videos of unseen subjects by leveraging few example images.

As shown in Fig. 3 (a), given the source image $I_{s_{i}}$ and the reference image $I_{r}$, the role of this stage is to predict the kinematic pose (rotation of limbs) and shape parameters, as well as the 3D mesh of each image. In this paper, we use the HMR [17, 18] as the 3D pose and shape estimator due to its good trade-off between accuracy and efficiency. In HMR, an image is firstly encoded into a feature with $\mathbb{R}^{2048}$ by a ResNet-50 [50] and then followed by an iterative 3D regression network that predicts the pose $\theta\in\mathbb{R}^{72}$ and the shape $\beta\in\mathbb{R}^{10}$ of SMPL [16], as well as the weak-perspective camera $K\in\mathbb{R}^{3}$. SMPL is a 3D body model that can be defined as a differentiable function $M(\theta,\beta)\in\mathbb{R}^{N_{v}\times 3}$, and it parameterizes a triangulated mesh by $N_{v}=6,890$ vertices and $N_{f}=13,776$ faces with the parameters of a pose $\theta\in\mathbb{R}^{72}$ and a shape $\beta\in\mathbb{R}^{10}$. Here, the shape parameters $\beta$ are the coefficients of a low-dimensional shape space learned from thousands of registered scans, and the pose parameters $\theta$ are the joint rotations that articulate the bones via forwarding kinematics. With such process, we will obtain the body reconstructive estimations of each source image, $\{K_{s_{i}},\theta_{s_{i}},\beta_{s_{i}},M_{s_{i}}\}$ and those of reference image, $\{K_{r},\theta_{r},\beta_{r},M_{r}\}$, respectively.

Based on previous estimations, we first render a correspondence map and a weight index map for each source mesh $M_{s_{i}}$ and the reference mesh $M_{r}$ under the camera view of $K_{s_{i}}$ and $K_{r}$. Here, we denote the source weight index map, the source and the target correspondence maps as $W_{s_{i}}$, $C_{s_{i}}$ and $C_{t}$, respectively. In this paper, we use a fully differentiable renderer, Neural Mesh Renderer (NMR) [51]. We thereby project vertices of the source $V_{s_{i}}$ into a 2D image space by a weak-perspective camera, $v_{s_{i}}=\pi(V_{s_{i}},K_{s_{i}})$. Here, $\pi$ is the weak-perspective projective function. Then, we calculate the barycentric coordinates of each mesh face and obtain $f_{s_{i}}\in\mathbb{R}^{N_{f}\times 2}$. Next, we calculate the transformation flow $T_{s_{i}\to t}\in\mathbb{R}^{H\times W\times 2}$ by matching the correspondences between the source correspondence map with its mesh face coordinates $f_{s_{i}}$. Here $H\times W$ is the size of the image. By the same means, we obtain the transformation flow $T_{r\to t}$ of the reference correspondence map. We describe the procedure to obtain the transformation flow in Algorithm 1. Consequently, a foreground image $I^{ft}_{s_{i}}$ and a masked background image $I^{bg}_{s_{i}}$ are derived from masking the source image $I_{s_{i}}$ based on $C_{s_{i}}$. We randomly pick one of the masked backgrounds, denoted as $I_{bg}$, because all source images share the same background. Finally, we warp the visible textures of each source image $I_{s_{i}}$ to the desired condition by the transformation flow $T_{s_{i}\to t}$ and thereby obtain a synthetic image $I^{syn}_{t}$, as depicted in Fig. 3.

This stage synthesizes high-fidelity human images under the desired condition. More specifically, it 1) synthesizes the background image; 2) predicts the color of invisible parts based on the visible parts; 3) generates pixels of clothes, hairs, and others out of the reconstruction of SMPL.

Generator. Our generator works in a three-stream manner. One stream, named $G_{BG}$, works on the concatenation of the masked background image $I_{bg}$ and the mask obtained by the binarization of $C_{s_{i}}$ in the color channel to generate the realistic background image $\hat{I}_{bg}$, as shown in the top stream of Fig. 3 (c). The other two streams are the source identity stream, namely $G_{SID}$ and the transfer stream, namely $G_{TSF}$. $G_{SID}$ is a denoising convolutional auto-encoder that aims to guide the encoder to extract the features that are capable of preserving the source information. Together with the $\hat{I}_{bg}$, it takes the masked source foreground $I^{ft}_{s_{i}}$ and the correspondence map $C_{s_{i}}$ as its inputs and reconstructs source foreground image $\hat{I}_{s}$. $G_{TSF}$ stream synthesizes the final result, which receives the warped foreground by a bilinear sampler and the correspondence map $C_{t}$ as its inputs. To preserve the source information, such as texture, style, and color, we propose a novel Liquid Warping Block (LWB), as well as its advanced version, Attentional Liquid Warping Block (AttLWB), that links the source with the target streams. They blend the source features from $G_{SID}$ and fuses them into the transfer stream $G_{TSF}$, as shown at the bottom of Fig. 3 (c).

$G_{BG}$ and $G_{SID}$ have similar architectures with separate parameters and follow the structure of CycleGAN [52] with 6 residual blocks [53]. The details of kernel sizes and number of filters are illustrated in Fig. 5. $G_{TSF}$ is a combination of a ResNet and a U-Net [33], named ResUnet. For $G_{BG}$, we directly regress the final background image, $\hat{I}_{bg}$, while for $G_{SID}$ and $G_{TSF}$, we concretely generate an attention map $A$ and a color map $P$, as shown in Fig. 5. The final image can be obtained as follows:

Here, $\odot$ represents an element-wise multiplication. The total trainable parameters in the generator are $\theta_{G}=\{\theta_{BG},\theta_{SID},\theta_{TSF},\theta_{AttLWB}\}$, with respect to $G_{BG}$, $G_{SID}$, $G_{TSF}$ and AttLWB.

Discriminator. To push the discriminators to focus on different aspects of the generated images, such as the clothes on the human body and the face identity, we utilize a global-local content-orientation architecture. It consists of three sub-discriminators. The first one is a global discriminator, $D_{Global}$, which regularizes the entire generated $\hat{I}_{t}$ to be more realistic-looking. The rest two are a body discriminator $D_{Body}$ and a face discriminator $D_{Head}$, and they push the cropped body area and the head (face) parts of the generated $\hat{I}_{t}$ to be realistic-looking. All of them are conditional discriminators, and they take the generated images and the correspondence map $C_{t}$ as their inputs. We illustrate the details of our discriminators in Fig. 5. The total trainable parameters in the discriminators are $\theta^{D}=\{\phi_{Global},\phi_{Body},\phi_{Head}\}$.

Attentional Liquid Warping Block. One advantage of our proposed Liquid Warping Block (LWB) and Attentional Liquid Warping Block (AttLWB) is that it addresses the issue of multiple sources. For instance, in human motion imitation, the source images are multi-view inputs, and in the appearance transfer, different parts of garments come from different people. The different parts of features are aggregated into $G_{TSF}$ by their transformation flow independently. As shown in Fig. 4, we denote $X^{l}_{s_{1}}$ and $X^{l}_{s_{2}}$ as the feature maps extracted by $G_{SID}$ of different sources at the $l^{th}$ layer and $X^{l}_{t}$ is the feature map of $G_{TSF}$ at the $l^{th}$ layer. Each part of the source feature is warped by their transformation flow and aggregated into the features of $G_{TSF}$. We use a bilinear sampler (BS) to warp the source features $X^{l}_{s_{1}}$ and $X^{l}_{s_{2}}$ with respect to corresponding transformation flows, $T_{s_{1}\to t}$ and $T_{s_{2}\to t}$. The way to aggregate the warped source features into the global stream is the main difference between LWB and AttLWB.

LWB, as illustrated in Fig. 4 (a), directly uses an element-wise addition among all features and the fuses the global features as:

However, LWB will enlarge the magnitude of the features in the overlap area, and thereby result in artifacts. To address this, motivated by the attention architecture [22], we propose a more advanced Attentional Liquid Warping Block (AttLWB), as shown in Fig. 4 (c). It firstly learns similarities of the global features among all multiple source features, and then it fuses the multiple source features by the linear combination of the learned similarities and the multiple sources in feature space. Finally, to better propagate the source identity (style, color, and texture) into the global stream, we use the SPADE [23] to denormalize the feature map of $G_{TSF}$ with the fused source features to obtain the global stream, which could further improve the final result. We describe the entire procedures of AttLWB in Algorithm 2.

In this part, we will introduce the loss functions and how to train the whole system. For the body recovery module, we follow the network architecture and loss functions of HMR [17, 18]. Here, we use a pre-trained (off-the-shelf) SMPL estimator.

Note that our proposed Attentional Liquid Warping GAN is a unified framework for motion imitation, appearance transfer, and novel view synthesis. Therefore once we have trained the model on one task, it is capable of being applied to other tasks. These three tasks share the same training pipeline in our method, except for the way to sample the source the reference images. In motion imitation, we randomly sample $s_{n}+1$ images from each video with difference poses and set the first $s_{n}$ ones as the source images $\{I_{s_{1}},...,I_{s_{n}}\}$ and the other one as the reference $I_{r}$. In appearance transfer, we need to sample $s_{n}+1$ images with the same person identity wearing different clothes, while in novel view synthesis, we need to sample $s_{n}+1$ images of the same person under the different camera of views. In our experiments, we train a model for motion imitation and then apply it to appearance transfer and novel view synthesis.

The whole loss function of the generator contains four terms, which are perceptual loss [54], face identity loss, attention regularization loss, and adversarial loss.

Perceptual Loss. It regularizes the reconstructed source image $\hat{I}_{s_{i}}$ to the ground truth $I_{s_{i}}$ and pushes the generated target image $\hat{I}_{t}$ and the reference image $I_{r}$ to be closer in a VGG [55] feature subspace. Its formulation is given as follows:

Here, $f$ is a pre-trained VGG-19 [55] on ImageNet [56].

Face Identity Loss. It regularizes the cropped face from the synthesized target image $\hat{I}_{t}$ to be similar to that from the image of ground truth $I_{r}$, which pushes the generator to preserve the face identity. It is shown as follows:

Here, $g$ is a pre-trained SphereFaceNet [57].

Adversarial Loss. It pushes the distribution of synthesized images to the distribution of real images. We use a $LSGAN_{-110}$ [58] loss in a way like PatchGAN over all discriminators, $D_{Global}$, $D_{Body}$ and $D_{Head}$. They push the entire generated images, cropped body area, and head (face) parts to be realistic-looking. We denote the bounding box of head and body as $head^{xy}$ and $body^{xy}$ in the ground-truth $I_{r}$, respectively, and we calculate them by the projected vertices in the image space. $\hat{I}^{b}_{t}$, $I^{b}_{r}$ and $C^{b}_{t}$ are the cropped bodies from the generated image, the reference image and the correspondence map, based on bounding box of body, $body^{xy}$. $\hat{I^{h}_{t}}$, $I^{h}_{r}$ and $C^{h}_{t}$ are the corresponding cropped heads with respect to the bounding box of head, $head^{xy}$. We arrive at the total adversarial loss as follows:

Attention Regularization Loss. It regularizes the attention map $A_{t}$ and $A_{s_{i}}$ to be smooth and prevents them from saturating. Considering that there is no ground truth of attention map $A$ or color map $P$, they are learned from the resulting gradients of above losses. However, the attention masks can easily saturate to 1 which prevents the generator from working. To alleviate this situation, we regularize the mask to be closer to the silhouettes $S$ rendered from a 3D body mesh. Since the silhouettes is a rough map and it contains the body mask without clothes and hair, we addtionaly introduce a Total Variation Regularization [59] over $A$ to compensate the shortcomings of silhouettes. It is shown as:

For the generator, the full objective function is shown as follows, and $\lambda_{p},\lambda_{f}$ and $\lambda_{a}$ are the weights of perceptual, face identity and attention losses, respectively.

For discriminator, the full objective function is

Though we can train our model on a large dataset, to a certain degree, with diverse people and clothes, however, such a generator is still hard to be well-generalized to the inputs out of the domain of training set. After all, it is infeasible to build a universal dataset and generator to handle the diverse face identities, styles of clothes, and backgrounds. To improve the generalization, inspired by the SinGAN [24] and the Meta-learning [25, 49, 10, 46], we apply the one/few-shot adversarial learning to push the network to focus on each individual by several steps of fast personal adaptation. In real application scenarios, the user might only provide a little number ($s_{n}$) of their photos with different views or poses, and in an extreme case, there is only one image accessible. In this paper, we focus on the setting where there are no more than eight images ($s_{n}\leq 8$) [25] available in the testing phase.

Specifically, we first train our model, including a generator and a discriminator, on a combined large dataset, and consequently obtain the generator’s pre-trained parameters, $\theta^{M}_{G}$, and the discriminator’s pre-trained parameters, $\theta^{M}_{D}$. Then, for each specific person $P_{i}$ with $s_{n}$ images, we learn the person-specific generator $\theta^{P_{i}}_{G}$ and discriminator $\theta^{P_{i}}_{D}$ from the $s_{n}$ images by fine-tuning the pre-trained model. This process is called one/few-shot personalization. To further push the generator from the pre-trained $\theta^{M}_{G}$ to the person-specific $\theta^{P_{i}}_{G}$, we discard the pre-trained parameters of the discriminator $\theta^{M}_{D}$, and we train the person-specific discriminator $\theta^{Pi}_{D}$ from scratch. The overall loss functions in the personalization phase are similar to that in the training phase, except for the adversarial loss. Since there are only a few images ($s_{n}\leq 8$), to avoid overfitting and reduce the time consumption of each iteration in personalization, we only use the global discriminator.

After we conduct personalization, the person-specific generator can be applied to all three tasks. The difference lies in the transformation flow computation, due to the different conditions of various tasks. The remaining modules, Body Mesh Recovery and Liquid Warping GAN (Attentional Liquid Warping GAN) are all the same. The followings are the details of each task of the Flow Composition module in the testing phase.

Motion Imitation. We firstly copy the value of pose parameters of the reference $\theta_{r}$ into that of the source and get the synthetic parameters of SMPL, as well as the 3D mesh, $M_{t}=M(\theta_{r},\beta_{s})$. Next, we render a correspondence map of the source mesh $M_{s}$ and that of the synthetic mesh $M_{t}$ under a camera view $K_{s}$. Here, we denote the source and the synthetic correspondence map as $C_{s}$ and $C_{t}$, respectively. Then, we project the source vertices into the 2D image space by a weak-perspective camera, $v_{s}=\pi(V_{s},K_{s})$. Here, $\pi$ is the weak-perspective projective function. Next, we calculate the barycentric coordinates of each mesh face and have $f_{s}\in\mathbb{R}^{N_{f}\times 2}$. Finally, we calculate the transformation flow $T\in\mathbb{R}^{H\times W\times 2}$ by matching the correspondences between the source correspondence map with its mesh face coordinates $f_{s}$ and the synthetic correspondence map. It is shown in Fig. 6 (a).

Novel View Synthesis. Given a new camera view, in terms of a rotation $R$ and a translation $t$. We firstly calculate the 3D mesh under the novel view, $M_{t}=M_{s}R+t$. The consequential operations are similar to that of motion imitation. We render a correspondence map of the source mesh $M_{s}$ and that of the novel mesh $M_{t}$ under a weak-perspective camera $K_{s}$ and calculate the transformation flow $T\in\mathbb{R}^{H\times W\times 2}$, as depicted in Fig. 6 (b).

Appearance Transfer. We need to “copy” the clothes on the body from the reference image while keeping the head (face, eye, hair and so on) identity of the source. We split the transformation flow $T$ into two sub-transformation flows, source flow $T_{1}$ and referent flow $T_{2}$. We denote the head mesh as $M^{h}=(V^{h},F^{h})$ and the body mesh as $M^{b}=(V^{b},F^{b})$. Here, $M=M^{h}\cup M^{b}$. For $T_{1}$, We firstly project the head mesh $M^{h}_{s}$ of source into the image space and thereby obtain the silhouettes, $S^{h}_{s}$. Then, we create a mesh grid, $G\in\mathbb{R}^{H\times W\times 2}$. Next, we mask $G$ by $S^{h}$ and derive $T_{1}=G\odot S^{h}$. Here, $\odot$ represents an element-wise multiplication. For $T_{2}$, it is similar to that in motion imitation. We render the correspondence map of the source body $M^{b}_{s}$ and that of the reference $M^{b}_{t}$, denoted as $C^{b}_{s}$ and $C^{b}_{t}$, respectively. Finally, we calculate the transformation flow $T_{2}$ based on the correspondences between $C^{b}_{s}$ and $C^{b}_{t}$. We illustrate it in Fig. 6 (c).

iPER. To evaluate the performance of our proposed method of motion imitation, appearance transfer, and novel view synthesis, we build a new dataset with diverse styles of clothes in videos, named Impersonator (iPER) dataset. There are 30 subjects of different conditions of shape, height, and gender. Each subject wears different clothes and performs an A-pose video and a video with random actions. There are 103 clothes in total. The whole dataset contains 206 video sequences with 241,564 frames. We split it into training/testing set at a ratio of 8:2 according to the different clothes. All the clothes and 29% of the actors in the testing set do not appear in the training set. We illustrate the details of the iPER dataset in classes of actions, styles of clothes, weight, and height distributions of actors in Fig. 7. We show some samples in the first two rows of Fig. 8.

MotionSynthetic. We also make up a synthetic dataset, named MotionSynthetic, for the convenience of evaluation, especially for human appearance transfer and novel view synthesis, because we can synthesize the ground truth images with different views and wearing garments by the modification of meshes. This dataset borrows 24 human meshes from people snapshot [60] and 96 human meshes from MultiGarments [40]; thus, 120 meshes in total. All of these meshes with UV texture images have been registered in SMPL [16]. For each mesh, we choose a pose sequence from Mixamo and a background image from the Internet. Based on these materials (mesh, UV image, pose sequence, and background image), we render the synthetic images by NMR [51], resulting in 39,529 frames in total. We split it into training/testing set at a ratio of 8:2 according to the different meshes and illustrate some synthetic images in the 3rd rows of Fig. 8.

FashionVideo. It contains 500 training and 100 testing videos with a single female model wearing fashionable clothes [20]. Each video has around 350 frames. The clothes and textures are diverse, while there are few types of gestures, with only a few standard poses for the models. Also, this dataset lacks diversity in background, and all the backgrounds are black. We display some samples in the 4th row of Fig 8.

Youtube-Dancer-18. To further validate the effectiveness and generalization of our method, we evaluate our method on the in-the-wild internet videos, Youtube-Dancer-18 [10]. It consists of 18 videos, with people dancing, downloaded from Youtube, and each of them lasts from 4 to 12 minutes. We follow the setting with MetaPix [10] that we sample frames with 30 FPS and only use $s_{n}\leq 8$ frames from training sequences for personalization and then apply the evaluation on the testing sequences. Some samples are shown at the bottom of Fig. 8. It needs to be mentioned that we do not train the model in this dataset. We only sample $s_{n}$ frames for personalization and directly test on this dataset to evaluate the generalization over all methods.

We train our Attentional Liquid Warping GAN on a combined dataset consisting of the iPER, MotionSynthetic, and FashionVideo dataset and perform evaluations among these three datasets. To evaluate our methods’ generalization, we also perform tests on an additional Youtube-Dancer-18 dataset without training on it. We crop all images based on the bounding box of the human body, rescale the cropped images with keeping the original ratio of height and width, and then pad them into a $512\times 512$ resolution. We normalize the color space of all images to [-1, 1]. In our experiments, including the training and personalization phase, we use the Adam [61] based Stochastic Gradient Descent optimizer for both generators and discriminators. $\lambda_{p},\lambda_{f}$ and $\lambda_{a}$ are 10.0, 5.0 and 2.5, respectively.

i): In the training phase, we randomly sample $s_{n}+1$ images from each video and set the first $s_{n}$ ones as the source images $\{I_{s_{1}},...,I_{s_{n}}\}$, and the other one as the reference $I_{r}$. We fix $s_{n}=2$ and the mini-batch size to be 2. There are two training epochs. We fix the first quarter training session with a learning rate as 0.0001 and gradually decrease it to 0.00001 in the end.

ii): In the personalization and testing phase, $s_{n}$ could be flexible, and because of the memory limitation of the GPU devices, in our experiments, we set $s_{n}\in\{1,2,4,8\}$. Besides, $I_{r}$ lies in the set of source images $\{I_{s_{1}},...,I_{s_{n}}\}$. We fix the learning rate as 0.0001 and take $T=100$ steps for personalization.

Evaluation Metrics. We propose an evaluation protocol of the testing set of the iPER, MotionSynthetic, FashionVideo, and Youtube-Dancer-18 datasets, and it can indicate the performance of different methods in terms of different aspects. The details are listed in followings:

1): In each video with actor $P_{i}$, $\{I^{P_{i}}_{1},...,I^{P_{i}}_{t},...,I^{P_{i}}_{L}\}$, we select eight images as candidate images with different views, such as frontal, sideways or back. Here, $L$ is the number of frames.

2): We choose $s_{n}\leq 8$ images as sources, $\{I^{P_{i}}_{s_{1}},...,I^{P_{i}}_{s_{n}}\}$, from the eight candidate images for personalization. For a fair comparison with other methods [2, 1, 3, 36, 35], which only use a single source image, we separately report the results on $s_{n}=1$ (one-shot setting) and $2\leq s_{n}\leq 8$(few-shot setting).

3): After personalization, we perform self-imitation that each actor $P_{i}$ imitates actions from images of themselves, with $I^{P_{i}}_{t}$ as the reference image. We denote $\hat{I}^{P_{i}\to P_{i}}_{t}$ as the synthesized image referring to $I^{P_{i}}_{t}$. As for criterion, we use PSNR, SSIM [62], Learned Perceptual Similarity (LPIPS) [63], Body-CS and Face-CS to measure the similarities between $\hat{I}^{P_{i}\to P_{i}}_{t}$ and $I^{P_{i}}_{t}$.

Body-Cosine-Similarity (Body-CS): is the distance between the cropped person region of the synthesized image and that of the ground-truth image. In particular, it firstly uses a YOLOv3 [64] detector to get the person bounding box of the synthesized and ground-truth image. Then, we crop the person patches according to the bounding boxes. Finally, we use a pre-trained Person re-identification (ReID) model, OS-Net [65], to get the embedding features of the cropped person patches, and then we normalize the features and calculate the cosine similarity between the features to acquire the Body-CS.

Face-Cosine-Similarity (Face-CS): similar to Body-CS, it is the distance between the cropped face region of the synthesized image and that of the ground-truth image. Specifically, we firstly use an MTCNN [66] face detector to get the face bounding boxes of the synthesized and ground-truth images. Then, we crop the face regions according to the bounding boxes. Finally, we uses a pre-trained face recognition model [67], to get the embedding features of the cropped face patches, and then we normalize the features and calculate the cosine similarity between the normalized features to obtain the Face-CS.

4): We also conduct cross-imitation that an actor $P_{i}$ imitates actions from others, such as $P_{j}$. We denote $\{\hat{I}^{P_{i}\to P_{j}}_{1},...,\hat{I}^{P_{i}\to P_{j}}_{L}\}$ as a sequence of synthesized images referring to $\{I^{P_{j}}_{1},...,I^{P_{j}}_{L}\}$ and $\{I^{P_{i}}_{s_{1}},...,I^{P_{i}}_{s_{n}}\}$ as the sequence of real images. Since there is no ground-truth of synthesized images for the similarities metrics as mentioned above, here, we use a Fréchet Inception Distance (FID) [68] to measure perceptual realism. It calculates the distance between the set of synthesized images and that of real images. We further propose the Fréchet Distance of a pre-trained ReID model, OS-Net [65], namely Body-FD and that of a face recognition model, namely Face-FD. We also collect $L$ consecutive frames from the actor $P^{i}$, denoted as $\{I^{P_{i}}_{1},...,I^{P_{i}}_{L}\}$, then calculate the Body-CS and Face-CS as aforementioned.

Quantitative Comparison with Other Methods under One-shot Setting. We compare the performance of our method with that of existing methods, including PG2 [2], SHUP [1], DSC [3], DIAF [36] and PATB [35]. We train all these methods on a combined dataset with the iPER, MotionSynthetic, and FashionVideo dataset and apply the evaluation protocol with the one-shot setting mentioned above to these methods. We report the results in Table I, and our method outperforms others on all the metrics except SSIM, for which a higher numerical value does not necessarily mean a better quality of an image as reported in [63].

Quantitative Comparison with Other Methods under Few-shot Setting. We compare the performance of our method with pix2pixHD [27], SPADE [23], MetaPix pix2pixHD and MetaPix SHUP [10] under this setting. Here, we report the results on the Youtube-Dancer-18 dataset with the number of source images $s_{n}$ being $2$ in Table II and our method outperforms others.

Qualitative Comparison. Besides, we also analyze the generated images and make comparisons between ours and the above methods. From Fig. 9, we find that 1) the above methods that use 2D pose-guided inputs change the body shape of the source. For example, in the $2^{nd}$ row of Fig. 9, the scenario is a tall person imitating motion from a short person, and baseline methods change the height of the source body. However, our method is capable of keeping the body shape unchanged because our method disentangles the pose and the personalized shape of each actor. 2) In the light of our proposed AttLWB (LWB) and face identity loss, our method is more powerful in terms of preserving source identities, such as the face identity and cloth details of source than other methods, as shown in the $1^{st}$ and $2^{nd}$ row of Fig. 9. 3) Our method also produces high-fidelity images in the cross-imitation setting (imitating actions from others), which we illustrate in Fig. 10. As we can see in Fig. 10, the face identity, and clothes details, in terms of texture color and style, are preserved well. It shows that our method can achieve decent results in cross imitation even when the reference image comes from the Internet, which is out of the domain of our training dataset.

It is worth emphasizing that once the model has been trained, it can directly be applied in three tasks, including motion imitation, appearance transfer, and novel view synthesis. We conduct the experiments on the iPER dataset.

Evaluation Metrics. In the iPER dataset, subjects might wear different clothes, and we sample the same person’s images with different clothes as the source and the reference image. We use aforementioned PSNR, SSIM [62], LPIPS [63], Body-CS and Face-CS as the metrics.

Quantitative Results. We report the results of our methods with LWB and AttWLB on the iPER dataset in Table III. The results show that Attentional Liquid Warping Block (AttLWB) is slightly better than the LWB.

Qualitative Results. We randomly pick some examples displayed in Fig. 11. The face identity and clothes details, in terms of texture, color, and style, are preserved well by our method. It demonstrates that our method can achieve decent results in appearance transfer, even when the reference image comes from the Internet and is out of the domain of the iPER dataset, such as the last five columns in Fig. 11.

Evaluation Metrics. As for data in the iPER dataset, we have videos containing different views of a certain subject performing A-pose, and in the MotionSynthetic dataset, we render A-pose images with 3D meshes from different viewpoints. Thus, we obtain images of the same person in different views. For evaluation, we use PSNR, SSIM [62] and LPIPS [63] as the metrics.

Quantitative Results. In Table IV, we report the results of our methods AttLWB and that of other state-of-the-art methods, including AppFlow [41] and MV2NV [69], on the iPER and MotionSynthetic datasets based on the above evaluation metrics. The results show that our method outperforms other methods.

Qualitative Results. We randomly sample source images from the testing set of the iPER dataset and change the views from $30^{\circ}$ to $330^{\circ}$. The results are illustrated in Fig. 12. Our method is capable of predicting reasonable content of invisible parts when switching to other views and keep the source information, in terms of face identity and clothes details, even in the self-occlusion case, such as the middle and bottom rows in Fig. 12. Through Fig. 12, we can see that 1) even when the subjects have large motion deformation, such as the case in the $1^{st}$ row of Fig. 12, results of our method can keep the logo details of clothes. 2) The $2^{nd}$ row shows the results when the subjects have self-occlusion. 3) Our method can also handle cases with complex background as the $3^{rd}$ row in Fig. 12 shows. 4) The $4^{th}$ row of Fig. 12 shows cases in which subjects wear a long dress and have fluffy hair. 5) The $5^{th}$ row of Fig. 12 is the case with complex clothes texture.

We perform the ablation studies of with/without personalization to verify the effectiveness of personalization. Besides, we also analyze the effect of hyper-parameters, including the number of source images $1\leq s_{n}\leq 8$ and that of steps $T$ for personalization. Since we only use the Youtube-Dancer-18 dataset in the testing phase, it is reasonable to evaluate the generalization of our methods of with/without personalization on this dataset. Here, we use self-imitation evaluation metrics, as mentioned above.

With/Without Personalization. We conduct comparative experiments with and without personalization in our methods. Here, we fix the $s_{n}=2$ and $T=100$ in the phase of personalization. Table V shows that our method with personalization could achieve $1.0421$ higher in PSNR, $0.0599$ higher in SSIM, and $0.039$ lower in LPIPS than that without personalization on the Youtube-Dancer-18 dataset. Furthermore, we display some example results in Fig. 13, where the left-column two roles are the source images from the Youtube-Dancer-18 dataset, and the right-column two roles are the cartoon images from the Internet. We find that with personalization, 1) our method could keep the color style of the background unchanged, as shown in the $1^{st}$ top left of Fig. 13; 2) our method is capable of preserving the logo details in the source clothes, as depicted in the $2^{nd}$ bottom left of Fig. 13; 3) our method is more powerful in the generalization, even when the source images are cartoon style, as illustrated in the right column of Fig. 13. These demonstrate that personalization indeed plays a significant role in improving the generalization of our system.

Number of Source Images $s_{n}$. In our system, we adopt a few source images $1\leq s_{n}\leq 8$ for personalization, and we will analyze the impacts of $s_{n}$ to the final results. Here, we fix the number of steps to $T=t\in\{10,50,100,150,200\}$ respectively for personalization and list the PSNR with different $s_{n}\in\{1,2,4,8\}$ in Fig. 14. It shows that the performance grows with an increase of $s_{n}$ when $T$ is large enough. The reason for the performance increase is due to the increase of the invisible textures. However, it is worth noticing that when $T$ is small with respect to a large $s_{n}$, in the case of $T=10$ and $s_{n}=8$, the performance decreases. The reason might be that when $T$ is small, it is too hard for the network to fit those too many source images.

Number of Steps $T$ for Personalization. In the real application, we should take the number of steps $T$ into consideration because more steps will take more time. It is necessary to consider the trade-off between performance and overhead time for personalization. We set $s_{n}\in\{1,2,4,8\}$ and list the performance with different $T$ for personalization in Fig. 14. From Fig. 14, we can see that the performance saturates at around 150 steps.

In summary, based on the above analysis, we recommend that in the stage of personalization, finetuning around 100 steps should be enough, and if the time for personalization is limited, it would be better to use fewer source images.

In our methods, we apply a perceptual loss $L_{p}$, a face identity loss $L_{f}$, an attention regularization loss $L_{a}$, and an adversarial loss $L^{G}_{adv}$ (with global, body and head adversarial loss in details) to the full training loss functions. To validate the effectiveness of each term, we perform the ablation studies of the different loss functions. From Table VI, we can see that the model with the full loss would have the best performance. Besides, with the addition of $L_{f}$ and $L^{G}_{adv}$, the performance increases compared with that of the trial with only $L_{p}$.

To verify the impact of our proposed Attentional Liquid Warping Block (AttLWB), we design some baselines with the ways mentioned above to propagate the source information, including input concatenation, texture warping, and feature warping. The body recovery, flow composition modules, the basic network architectures, and all loss functions are the same except for the propagating strategies among our method and other warping baselines. Here, we denote early concatenation, texture warping, and feature warping, as $W_{C}$, $W_{T}$, and $W_{F}$, respectively. Also, we denote the $s_{n}$ source images as $\{I_{s_{1}},...,I_{s_{n}}\}$, their corresponding conditional inputs as $\{C_{s_{1}},...,C_{s_{n}}\}$ and their corresponding feature maps as $\{X^{l}_{s_{1}},...,X^{l}_{s_{n}}\}$ at the $l^{th}$ layer, respectively. The reference conditional inputs are $C_{t}$. The transformation flow of each source image to the reference is $T_{s_{i}\to t}$. We list the details of all warping baselines in followings:

Input Concatenation $W_{C}$. It directly concatenates all source images, their corresponding conditional inputs, as well as the reference conditional inputs, and then feeds them into the $G_{TSF}$ network, as shown in Fig. 2 (a).

Texture Warping $W_{T}$. Based on each transformation flow $T_{s_{i}\to t}$, we warp each source image $s_{i}$ to the reference condition, average the pixels of overlap regions, and synthesize an initial image. Then, we feed it into the $G_{TSF}$ network and generate the final image, as shown in Fig. 2 (b).

Feature Warping $W_{F}$. Instead of warping the source information in the image space, it propagates the source information in the feature space, based on the transformation flow. As mentioned above, we firstly obtain the warped feature $X^{l}_{s_{i}\to t}$ by using a bilinear sampler (BS) to warp each source feature $X^{l}_{s_{i}}$ concerning the corresponding transformation flow $T_{s_{i}\to t}$. According to the ways to aggregate the global feature $X^{l}_{t}$ from multiple warped source features $\{X^{l}_{s_{1}\to t},...,X^{l}_{s_{n}\to t}\}$, we can specifically subdivide them into the followings:

1. 1.

   Attention $W^{Att}_{F}$ (ours) is shown in Algorithm 2.

2. 2.

   Add-Aggregation $W^{A}_{F}$ (ours). It is the first version of our proposed Liquid Warping Block(LWB) [21], as shown in the Fig. 4 (a) and Equation (2).

3. 3.

   Mean-Aggregation $W^{M}_{F}$. Directly adding the warped features will enlarge the magnitude of the features in the overlap area and thereby results in artifacts. A naive way is to average all the warped features, shown as follows.

   $\displaystyle\widehat{X}_{t}^{l}$

   $\displaystyle=\frac{1}{s_{n}}\sum_{i=1}^{s_{n}}X^{l}_{s_{i}\to t}+X_{t}^{l}.$

   (9)

4. 4.

   Add-Soft-Gate $W^{A\odot}_{F}$. The warped feature might introduce the misalignment problem, and to address it, Dong et al. [11] utilizes a gated convolution to control the transformation degree. We firstly add all the warped features, then utilize a gated convolution, as shown in Equation (10). Here, $g$ is a function with two-convolution layers followed by a Sigmoid activation and $g(X_{t}^{l})\in[0,1]$. $\odot$ represents the element-wise multiplication.

   $\displaystyle\widehat{X}_{t}^{l}$

   $\displaystyle=g(X_{t}^{l})\odot\sum_{i=1}^{s_{n}}X^{l}_{s_{i}\to t}+X_{t}^{l}.$

   (10)

5. 5.

   Mean-Soft-Gate $W^{M\odot}_{F}$. It firstly averages all the warped features and following steps are the same with $W^{A\odot}_{F}$. The formulation is shown as follows:

   $\displaystyle\widehat{X}_{t}^{l}$

   $\displaystyle=g(X_{t}^{l})\odot\frac{1}{s_{n}}\sum_{i=1}^{s_{n}}X^{l}_{s_{i}\to t}+X_{t}^{l}.$

   (11)

We conduct a user study, with 64 volunteers, to assess the quality of the generated videos and compare the performance of the warping strategies mentioned above. Participants are shown 17 groups of videos with 7 videos generated by 7 warping strategies respectively in random order in each group. Besides, the shared source image and reference video of each group is also shown to the participants for reference. Participants are asked to choose the best video considering the quality of the face, clothes texture, figure pose, and background. Finally, 64 responses are collected, and the results are shown in Fig. 15. As we can see that our proposed AttLWB and AddLWB have the best appraise, scoring $41.73\%$ and $20.04\%$, respectively, far higher than others.

Our method could produce the results with different image resolutions, ranging from $256\times 256$, $512\times 512$, $1024\times 1024$ to $1920\times 1920$. Here, we benchmark the running time of our system in different image resolutions. Since a high resolution needs more memory allocation of the GPUs device, we perform all the tests on a Tesla V100S-PCIe-32G GPU with the Intel Xeon(R) E5-2620 2.10GHz CPUs. The image resolution of the source images is $4032\times 3024$, and that of the reference video with 165 frames is $1920\times 1080$. In Fig.16, we separately report the running time of preprocessing, personalization and inference, when synthesizing different resolutions, respectively. From Fig.16, we can see that the higher resolution consumes more running time, especially in the personalization and inference.

There are three main types of failure cases of our methods. The first one, as shown in the $1^{st}$ row of Fig. 17, is that source image contains a large area of self-occlusion, which introduces an ambiguity in textures and thereby results in a bad synthesized image. The second occurs when the Body Recovery Module fails and could not accurately estimate the pose parameters, as illustrated in the $2^{nd}$ row of Fig. 17. The rest is when the background inpaintor $G_{BG}$ fails, as shown in the $3^{rd}$ row of Fig. 17.

In addition, there are still some limitations of our system, 1) it cannot imitate the motions of hands and facial expressions from the reference images, since the 3D body parametric SMPL [16] used in our system does not contain the articulated hands and expressive face; 2) also, it cannot animate the large-motion body with too loose clothing like the skirt or evening dress; 3) it is affected by the different lighting environments among sources.

Therefore, for a better result, the input source images need to follow these guidelines:

• •

  They share the same static background without too complex scene structures. If possible, we recommend using the actual background.

• •

  The person in the source images holds an A-pose for introducing the most visible textures.

• •

  It is recommended to capture the source images in an environment without too much contrast in lighting conditions and lock auto-exposure and auto-focus of the camera.