Human Pose Transfer with Augmented Disentangled Feature Consistency

Our designed Pose Feature Encoder consists of a Pose Estimator network, a Keypoint Amplifier block, and a Pose Refiner network. Given a RGB image $\bm{x}\in\mathbb{R}^{3\times H\times W}$ of height $H$ and width $W$, the pre-trained Pose Estimator aims at extracting pose information $P(\bm{x})$ from the image $\bm{x}$. Similar to (Cao et al., 2017), the extracted pose information contains the downsampled keypoint heatmaps $\bm{h}\in\mathbb{R}^{18\times\frac{H}{8}\times\frac{W}{8}}$ and the part affinity fields $\bm{p}\in\mathbb{R}^{38\times\frac{H}{8}\times\frac{W}{8}}$. The keypoint heatmaps $\bm{h}$ store the heatmaps of 18 body parts, and the part affinity fields $\bm{p}$ store the location and orientation for heatmaps of body parts and background, which has 38 ($=(18+1)\times 2$) channels.

As the pre-trained pose estimator (OpenPose (Cao et al., 2019) in our implementation) is pre-trained on COCO keypoint challenge dataset (Lin et al., 2014), when applied on Mixamo-Pose and EDN-10k datasets with different distributions, more noise is made on keypoint heatmaps $\bm{h}$. To reduce the interference of noise, we apply a softmax function with a relatively small temperature $T$ (e.g., 0.01) as the Keypoint Amplifier to denoise the extracted keypoint heatmaps by increasing the gap between large and small values in the heatmaps and obtain the amplified heatmaps $\bm{h}^{\prime}$ by

As shown in Figure 2, by employing Keypoint Amplifier on the input heatmaps, the small probability, e.g., 0.2, will be squeezed to almost 0.0. On the contrary, the large probability, e.g., 0.8, will be squeezed to almost 1.0. Without the Keypoint Amplifier, the generator may still synthesize blurry limbs for the low probability areas and twist the generated person.

Finally, the Pose Refiner takes both the part affinity fields $\bm{p}$ and the amplified keypoint heatmaps $\bm{h}^{\prime}$ and produces the encoded pose feature vector $M(\bm{x})$. In this way, the pose information extracted from the Pose Estimator can be refined, and the influence caused by different limb ratios and/or camera angles and distances can be reduced.

While the Pose Feature Encoder is not capable of extracting static information, the static information, including background, personal appearance, etc., from another image $\bm{x}^{\prime}$, is captured automatically by another module with the help of the full objective function. Named as Static Feature Encoder, this module extracts only static features $S(\bm{x}^{\prime})$ from $\bm{x}^{\prime}$.

Given pose features $M(\bm{x}_{\mathrm{s}})$ extracted from a source image $\bm{x}_{\mathrm{s}}$ and static features $S(\bm{x}_{\mathrm{t}})$ extracted from a target image $\bm{x}_{\mathrm{t}}$, the Image Generator outputs the synthesized image $\bm{x}_{\mathrm{syn}}$ by

It is noted that many existing methods (e.g., (Chan et al., 2019)) attempt to learn pose-to-image or pose-to-appearance mapping solely via its generator. In that case, the generator has to learn three different functionalities: 1) memorizing the state information of the target person, 2) extracting representative pose features, and 3) combining the static and pose information to synthesize the target person image with the desired pose. Even though the generator can memorize the state information $\bm{x}_{\mathrm{t}}$ of the target person perfectly, once the desired pose $\bm{x}_{\mathrm{s}}$ is very different from the poses in the training dataset (e.g., the distance from the camera, skeleton scale from different persons, and occlusions), it is too difficult for the generator to achieve the above second and third functionalities at the same time. The results of Pix2Pix (Isola et al., 2017) and Everybody Dance Now (EDN) (Chan et al., 2019) in Section 4 also validate their disadvantages. DFC-Net, instead, decomposes the above three functionalities into three network modules, including the pose feature encoder, static feature encoder, and image generator, and thus enables the reconstruction’s quality improvement.

We employ an Image Discriminator (D) in an adversarial learning way to ensure the synthesized image $\bm{x}_{\mathrm{syn}}$ borrows the pose and static information from the source and target images ($\bm{x}_{\mathrm{s}}$ and $\bm{x}_{\mathrm{t}}$) separately. As both the source and target images during training contain the same person with the same appearance and background, they share almost the same static features, i.e., $S(\bm{x}_{\mathrm{t}})\simeq S(\bm{x}_{\mathrm{s}})$. Therefore, the output of the model $\bm{x}_{\mathrm{syn}}$ can also be treated as a reconstruction of the source image $\bm{x}_{\mathrm{s}}$, as the synthesized images contain the same pose features $M(\bm{x}_{\mathrm{s}})$ as the source image. This inspires us to resort to the conditional generative adversarial network (cGAN) (Isola et al., 2017), where the Image Discriminator attempts to discern between the real sample $\bm{x}_{\mathrm{s}}$ and the generated image $\bm{x}_{\mathrm{syn}}$, conditioned on the pose features $M(\bm{x}_{\mathrm{s}})$ extracted from the source image. That is, the Image Discriminator attempts to fit $D(\bm{x}_{\mathrm{s}},M(\bm{x}_{\mathrm{s}}))=1$ and $D(\bm{x}_{\mathrm{syn}},M(\bm{x}_{\mathrm{s}})))=0$. The adversarial loss is described as follows:

where

We enhance the Image Discriminator with a multi-scale discriminator $D=(D_{1},D_{2})$ (Wang et al., 2018b) and include the discriminator feature matching loss $\mathcal{L}_{\mathrm{fm}}$ in our objective. The feature matching loss is a weighted sum of feature losses from 5 different layers of the Image Discriminator, calculated by $L_{1}$ distance between the corresponding features of $\bm{x}_{\mathrm{s}}$ and $\bm{x}_{\mathrm{syn}}$.

In order to increase the training stability and improve the synthesized image quality, we also add the perceptual loss $\mathcal{L}_{\mathrm{per}}$  (Johnson et al., 2016) based on a pre-trained VGG network (Simonyan and Zisserman, 2014a).

The above adversarial training losses aim at penalizing discrepancy between the synthesized and source images directly in the raw image space. To improve the accuracy and robustness of the pose transfer results, we also introduce two disentangled feature consistency losses in terms of pose and static features to ensure the synthesized person looks like the target person and behaves as the source person separately. The pose consistency loss $\mathcal{L}_{\mathrm{mc}}$ measures the differences between the synthesized and source images in the pose feature space, and the static consistency loss $\mathcal{L}_{\mathrm{sc}}$ measures the differences between the synthesized and target images in the static feature space. They are both $L_{1}$ distances between the outputs from the corresponding encoders, formally defined as

Through disentangling the pose feature from the source images, we find that images with different persons can also be passed into the training process as the source images $\bm{x}_{\mathrm{s}}$ to improve the generalization ability of our model. Hence, we introduce a novel data augmentation method that extends the training dataset with the images of different persons, referred to as the support set, providing many kinds of unseen poses. Note that the subjects in support set can be arbitrary and are different from the primary training dataset, so the ground-truth images with the target person performing the pose of the source person are not available at all. As a result, the corresponding losses $\mathcal{L}_{\mathrm{adv}}^{+},\mathcal{L}_{\mathrm{per}}$ and $\mathcal{L}_{\mathrm{fm}}$ for the support set are not applicable, and we optimize relevant objective terms, which are defined by

where the weights $\lambda_{\mathrm{adv}},\lambda_{\mathrm{mc}},\lambda_{\mathrm{sc}}$ are the weights for each loss.

By bringing all the objective terms together, we train all components jointly except for the Pose Estimator to minimize the full objective $\mathcal{L}_{\mathrm{full}}$ below.

where $\lambda_{\mathrm{adv}},\lambda_{\mathrm{fm}},\lambda_{\mathrm{per}}$ are set to 1, 10, 10 following Pix2Pix (Isola et al., 2017) and Everybody Dance Now (EDN) (Chan et al., 2019), while $\lambda_{\mathrm{mc}},\lambda_{\mathrm{sc}}$ are set to 0.1, 0.01 by grid search. We set the $\lambda_{\mathrm{sc}}$ to 0.01 comparing to the $\lambda_{\mathrm{mc}}$ to balance the $\mathcal{L}_{\mathrm{mc}}$ and $\mathcal{L}_{\mathrm{sc}}$.

We built Mixamo-Sup as a support set for data augmentation to boost the generality of DFC-Net and organized two datasets, Mixamo-Pose and EDN-10k, to verify the effectiveness of the proposed DFC-Net for human pose transfer.

• •

  EDN-10k: We processed and tailored the dataset released by (Chan et al., 2019) to build the EDN-10k dataset. The original dataset consists of five long target videos, each lasting from 8 minutes to 17 minutes, and split into the training and test set. We chose the first four subjects since subject five only performed less complex dance poses. In each video of the original dataset, a different subject performed a series of different motions, and the camera was fixed to keep the background unchanged. We chose four subjects from the original dataset, uniformly sampled 10k frames as the training set and 1k frames as the test set for each subject, Since the original images have a large resolution of $1024\times 512$ and most areas are the fixed background, we cropped all frames to the middle $512\times 512$ square areas and resized them to $256\times 256$.

• •

  Mixamo-Pose: We randomly chose 4 characters, Andromeda, Liam, Remy, and Stefani, with 30 different pose sequences from Mixamo. To render the 3D animations into 2D images, we loaded each character performing each pose sequence on a white background into Blender (Blender Online Community, 2018), placed two cameras in front of and behind the character, and took the images. We centered the characters in the images according to their keypoints and resized them to 256$\times$256. Mixamo-Pose were split into training and test sets. For each character, the training set contains 1488 images with 15 poses, and the test set contains 1185 images with 15 other poses.

• •

  Mixamo-Sup: For data augmentation, we built a support set by rendering 15684 images of six new characters from Mixamo (Adobe Systems Inc., 2018) with another 15 unseen poses in the same way as Mixamo-Pose. Since it is unnecessary to contain the same person as the target person image, DFC-Net leveraged the support set as the source person images $x_{s}$. When training on both EDN-10k and Mixamo-Pose, we use Mixamo-Sup as the support set. Note that Mixamo-Sup has a totally different distribution from EDN-10k but still gains a huge improvement shown in Section 4.3.

Note that the experiments on EDN-10k only include the pose transfer on the same person because the ground truth of different people carrying the same pose are unavailable. For Mixamo-Pose, since we can manipulate different characters to do the same action, the experiments include the pose transfer both on the different people, e.g., transfer the unseen pose of Liam to Andromeda, and the same person, e.g., transfer the unseen pose of Andromeda to herself.

We compared our DFC-Net with the following competitive baselines:

• •

  Nearest Neighbors (NN): For each source person image $\bm{x}_{\mathrm{s}}$, we chose the image $\bm{x}^{\prime}$ in the training set $\mathcal{D}_{\mathrm{tr}}$ with the lowest mean square error (MSE) between the pose information $P(\bm{x}_{\mathrm{s}})$ and $P(\bm{x}^{\prime})$ as $\bm{x}_{\mathrm{syn}}$.

  (10)

  $\displaystyle\bm{x}_{\mathrm{syn}}$

  $\displaystyle={\arg\min}_{\bm{x}^{\prime}\in\mathcal{D}_{\mathrm{tr}}}{\left\|P(\bm{x}_{\mathrm{s}})-P(\bm{x}^{\prime})\right\|}^{2}_{2}.$

  The pose information was extracted by the same Pose Estimator as in our method.

• •

  Pose-guided Methods: We chose CycleGAN (Zhu et al., 2017), Pix2Pix (Isola et al., 2017) and Everybody Dance Now (EDN) (Chan et al., 2019) as the baselines. They all took the skeleton images as input instead of the original images, and we employed a pre-trained pose estimator (Cao et al., 2017) to extract keypoints, used OpenCV (Bradski, 2000) to connect pairs of keypoints with different colors to generate the skeleton images. To ensure fair comparisons, the face GAN and face keypoint estimator in EDN were not adopted in our implementation, as they are independent components and can be seamlessly adopted by other learning-based baselines.

• •

  Spatial Transformation Methods: We selected Liquid Warping GAN (LWG) (Liu et al., 2019a), Monkey-Net (MKN) (Siarohin et al., 2019a) and First Order Motion Model (FOMM) (Siarohin et al., 2019b). LWG calculates the flow fields with additional 3D human models and integrates the human pose transfer, appearance transfer, and novel view synthesis into one unified framework. MKN and FOMM are both object-agnostic frameworks using learned keypoints to generate image animation in a self-supervised fashion.

We evaluated the quality of the synthesized images with three commonly used metrics:

• •

  MSE: The mean squared error between the values of pixels of synthesized images and ground-truth images. Lower MSE values are better.

• •

  PSNR: The peak signal-to-noise ratio, which provides an empirical measure of the quality of synthesized images regarding ground-truth images. Higher PSNR values are better.

• •

  SSIM: Structural similarity (Wang et al., 2004), which is another perceptual metric that quantifies the quality of synthesized images given ground-truth images and focuses more on structural information (e.g., light). Higher SSIM values are better.

• •

  IS: Inception Score (Salimans et al., 2016) is a metric for estimating the quality of the synthetic images based on the Inception-V3 model (Szegedy et al., 2016). Higher IS values are better.

• •

  FID: Frechet Inception Distance (Heusel et al., 2017) is also an Inception-V3-based metric to evaluate the synthetic images according to the statistics of the synthetic images. Lower FID values are better.

We calculated the average scores among all pairs of synthesized and ground-truth images on the test set. On Mixamo-Pose, for each character as the target person, we reported the average metrics of 4 different characters as the source person. While on EDN-10k, we reported metrics of the task on the same person for every subject.

Tables 2, 2,  4, 4, and 6 depict results in terms of MSE, SSIM, PSNR, IS, and FID on the EDN-10k dataset. Table 6 provides comparisons on EDN-10k in terms of average results of the above five metrics over four subjects. The experimental results validated the advances of DFC-Net for real images:

• •

  Our method consistently outperformed all the baseline methods on all subjects. When synthesizing real person images, the most significant result is that on Subject1, our method achieved 45.2043 MSE while NN, CycleGAN, Pix2Pix, EDN, LWG, and MKN only got MSE values of 54.9448, 64.1959, 58.3633, 56.3549, 51.6246 and 48.1603 in Table 2, which indicates the images generated by our method have clear details. From Tables 4 and 6, DFC-Net also excelled other baselines for all four subjects according to the Inception Score (IS) and Frechet Inception Distance (FID). As shown in Table 6, our method also achieved the highest average SSIM of 0.8269 for all subjects, while no other methods except MKN got SSIM score greater than 0.8, which shows that our synthesized images are more realistic and suitable for the human visual system.

• •

  Secondly, we noticed CycleGAN has the worst results, which are 64.1959, 77.4336, and 70.3681 for MSE scores, and 0.5256, 0.4911, and 0.5869 for SSIM scores on Subject1, Subject2, and Subject3 in the Tables 2 and  2 respectively. We argue that CycleGAN is better at transferring the color or style for images from two domains rather than changing the geometry of the images, such as recovering the human appearance from the human skeleton because CycleGAN aims to learn a mapping from unpaired images directly. This property of CycleGAN is also supported by its inferior PSNR scores compared with other methods.

• •

  We could observe that NN can achieve low scores of MSE in Table 2. Since there are a lot of training images, it is easier for NN to find an image whose motion is very close to the desirable motion. Moreover, the fixed background also makes NN have higher scores of SSIM and PSNR in Tables 2 and  4, while other methods have to learn to generate an accurate background. But the images generated by NN usually do not perform the desired motions, and do not have any temporal coherence in motion when the input is a motion sequence since the results only depend on the training set.

• •

  Moreover, we observed that EDN, LWG, MKN, and FOMM also provided good results, especially on MSE metric, e.g., 42.5696, 42.4368, 37.1098, and 37.3648 average values for all subjects in Table 2, comparing with other baselines. Taking Subject2 as an example, LWG, MKN, and FOMM provided the SSIM of 0.8445, 0.8375, 0.8503, and 0.8445 in Table 2 which are higher than the results of NN, CycleGAN, and Pix2Pix. The higher SSIM values show that LWG, MKN, and FOMM can synthesize images closer to the ground truths.

Tables 8, 8, 10, 10 and 12 respectively show the quantitative results of pose transfer in terms of MSE, SSIM, PSNR, IS, and FID on the Mixamo-Pose dataset. Table 12 provides comparisons on Mixamo-Pose in terms of average results of the above five metrics over four characters. We obtained the empirical results clearly demonstrated the effectiveness of DFC-Net on animation images:

• •

  As shown in Tables 8, 8,  10, 10 and 12, our DFC-Net outperformed all competing baselines regarding all five metrics on average again like the results on EDN-10k Notably, in terms of MSE, DFC-Net outperformed the second-best LWG by 0.8223 in Table 8. To be more concrete, the lowest MSEs of our DFC-Net indicated DFC-Net provided the most accurate motion transfer images with the shortest $L_{2}$ distance between the ground-truth images. For instance, our method provides 22.0763 of MSE on Remy comparing to the 27.1801, 28.2237, 24.0687, and 23.9229 from NN, CycleGAN, Pix2Pix and EDN in the Table 8. Similar results of IS and FID can also be observed in Tables 10 and 12. Furthermore, together with the highest scores of PSNR and SSIM, our DFC-Net could generate synthesized images with the most accurate motion transfer and the best image quality simultaneously.

• •

  Secondly, NN provided a relative good results on Andromeda but a higher MSE on Liam and Remy in the Table 8, since its performance depends on the training dataset in the extreme, and it can’t provide stable synthesized results. Moreover, in Tables 8 and  8, CycleGAN also can not perform well, with the results are 27.5520, 29.9436, and 28.2237 MSE, and 0.7205, 0.7154 and 0.7377 for SSIM scores on Andromeda, Liam, and Remy respectively. These results once again validated that it is difficult to learn a mapping from the skeleton images to human images directly using unpaired data.

• •

  Thirdly, from Tables 8, 8, 10, 10, and 12, Pix2Pix, EDN, and LWG delivered such superior performance comparing with NN and CycleGAN. because they aim to learn a paired mapping from the skeleton images to human images directly. Besides that, the skeleton images as their input have accurate motion information without any noise and make the learning process easier. In contrast, even without converting source person images to skeleton images, our method DFC-Net fills the gap between the original source person images and the skeleton images to some extent by introducing the keypoint amplifier, two consistency losses, and a support dataset for training. Without any pre-process steps, and thus the source person images contain much noise and redundant information, our method can still achieve more improvements over Pix2Pix and EDN according to all five metrics.

• •

  Compared to the results on EDN-10k, MKN and FOMM dropped their performance when they transferred the pose between different people on Mixamo-Pose. It is difficult for them to extract keypoints features without a pre-trained pose estimator when the source person was not in the training dataset. For example, in Table 8, we can see that the SSIM score of MKN on Andromeda is 0.7076 compared with 0.7357, 0.7784, 0.7817 and 0.7726 from NN, Pix2Pix, EDN, and our method.

In Tables 14 and 14, the baseline (the first row) is the results of our model without the keypoint amplifier, the consistency losses and the support dataset. This baseline provided the worst results of the three metrics, e.g., the MSE is 26.4595 in Table 14. The results in the second row versus those in the first row showed that the keypoint amplifier strengthened the performance of pose transfer, such as promoting the SSIM of the baseline from 0.6595 to 0.6676 in Table 14. Even with the consistency losses and support set (rows 6 and 7), it continuously reduced the interference of the noise on the keypoint heatmaps. These results validated that the keypoint amplifier can filter out the real keypoint locations and reduce the interference of the noise on the keypoint heatmaps.

We explored the effects of consistency losses $\mathcal{L}_{\mathrm{sc}}$ and $\mathcal{L}_{\mathrm{mc}}$ on the task of pose transfer. Considering the different results between the second row and the third row, the static feature consistency loss $\mathcal{L}_{\mathrm{sc}}$ boosted the pose transfer task and verified its validity. Moreover, the performance variations between the second row and the fourth row clearly evidenced the advantages of pose feature consistency loss $\mathcal{L}_{\mathrm{mc}}$. Together with these two feature consistency losses, the model achieved better results with MSE of 21.9509, PSNR of 34.8292, and SSIM of 0.8032, as shown in the fifth row of Table 14 The comparison among these cases showed that each consistency loss, whether the static one or the motion one, can enforce the consistency between the real image and synthesized images, thus improving the quality of the synthesized images.

It is noted that the static feature consistency was more useful than the motion one on EDN-10k, while we observed the opposite performance on Mixamo-Pose. For example, in the third and fourth rows of Table 14, the static feature consistency loss achieved 0.0104 improvements for SSIM, while the motion feature consistency loss provided a 0.0285 improvement. We considered that since the backgrounds of images in Mixamo-Pose were simply white (i.e., easy to learn), the static feature consistency could only boost the performance for personal appearance on Mixamo-Pose. On the contrary, the backgrounds were more complex in EDN-10k. Thus the static feature consistency was more effective on EDN-10k than on Mixamo-Pose.

We added the support set and utilized the augmented consistency loss $\mathcal{L}_{\mathrm{sup}}$ during the training. The comparisons among the final row and other rows notably indicated that the support set could improve the generalization ability and the robustness of our model, especially when the poses of the source person are closer to the poses in the support set. The support set provided more negative examples besides the original training set and could help the discriminator to form an accurate boundary, which could further strengthen the performance of the generator. We also noticed that the support set was more useful on EDN-10k, because EDN-10k was sampled from the video clips of subjects, and it did not contain much motion variance compared with Mixamo-Pose. Moreover, the limb ratios and the distances between the person and camera in the support set were also distinct from EDN-10k, which provided more valuable negative samples for EDN-10k.

To further verify the contribution of different pre-trained pose estimator backbones, we also conducted extensive experiments on Subject1 from EDN-10k and Liam from Mixamo-Pose with ResNet architectures including ResNet-18, ResNet-50, ResNet-101, and ResNet-152. For ResNet-18, we did not find any pre-trained model online, and thus we re-implemented (Cao et al., 2017) by replacing the VGG-19 with the ResNet-18 following the same training process as (Cao et al., 2017). For ResNet-50, ResNet-101, and ResNet-152, we directly use the pre-trained models from (Xiao et al., 2018). The results are shown in Tables 16 and 16. We can observe that when we use a larger pose estimator backbone, we usually get better performance according to all five metrics. For instance, on Subject1 from EDN-10k, the ResNet-152 achieved the best SSIM result of 0.7261 compared to other backbones. It is because a better pre-trained pose estimator backbone can provide more representative pose information and thus generate high-fidelity images. It suggests that our method still has the potential for improvement by combining more advanced pose estimation technologies. In this work, we mainly focus on how to add consistency in the feature space instead of directly using a better pose estimator backbone.

We also provided visualization results on EDN-10k in Figure 3 and cropped the results in Figure 4 with more details, and DFC-Net synthesized real person image results with better qualities, which clearly indicated the effectiveness of our method as well. For instance:

• •

  As shown in the second row of Figure 3, the difference between the source pose and the target pose is very large, and this posture involves changes in almost all parts of the human body. In that case, we can observe that DFC-Net can generate the correct human pose for the Subject2, when other methods like LWG, MKN, and FOMM can only synthesize distorted poses. Another example is in Figure 4, we observed that the synthesized images of our method contained more details, including the clear face, wrinkles on clothes, etc. The generated pose transfer image of Subject1 from DFC-Net has a more clear face structure, and the face orientation of the characters is also consistent with the target image, while MKN and FOMM failed to synthesize correct face orientation and hand poses.

• •

  Sometimes NN could generate images that are closer to the ground truths, e.g., the image of Subject3 at the third row of Figure 3, since the number of images in the training set increases from 1488 in Mixamo-Pose to 10000 in EDN-10k. But people could still easily determine that the poses of the generated images are different from the ground-truths.

• •

  The results produced by Pix2Pix and EDN were more blurry, and there was an aliasing effect on the edges of the subject in the Pix2Pix result. In addition, EDN showed better image results than Pix2Pix did. In the third row of Figure 3, we noticed that the image of Subject3, synthesized by Pix2Pix, had much noise in the subject’s hair. On the contrary, the result of EDN was smoother and more realistic, but they lacked many details compared with DFC-Net, especially the face part was still twisted.

As depicted in Figure 5, we employed multiple target persons and source persons with various poses that were different from those in the training set of the Mixamo-Pose. The first column is the target person image, and the second one is the source person image. Note that only our method DFC-Net takes the target person and source person images at the same time. CycleGAN, Pix2Pix, and EDN only take the corresponding skeleton image of the source person image as input and synthesize results. The last column is real images of the target person making the desired motion which are ground truth. We illustrate details as follows:

• •

  Firstly, compared to the aforementioned baselines, DFC-Net offered better transfer results and alleviated their drawbacks on synthesizing body parts, clothes, and faces, just to name a few. For instance, only DFC-Net generated the left hand of Liam at the first and second rows, while there was only the arm in the results of other baselines. When the target person and source person are not the same person, e.g., in rows 2-8, DFC-Net can still disentangle the pose information of the source person from the static information successfully and then synthesize high-quality results. In contrast, MKN and FOMM are unable to strip out the pose information fully, and thus the generated results are strange, e.g., the person in the result of FOMM is facing away in the fourth row.

• •

  Meanwhile, though the Pix2Pix and EDN could synthesize the target persons with the desired poses, some body parts (e.g., arms, hands) generated by Pix2Pix were severely corrupted. For example, images produced by EDN had stale colors and lots of noisy pixels in terms of clothes and faces, and it failed to capture the details of shoes in the 3rd row since the color of the shoes is white and is easily confused with the background.

• •

  Moreover, even though the image results of NN are more clear and not blurry, the NN method always made incorrect pose transfer since it can solely synthesize poses that existed in the training set. Besides that, NN can not synthesize images with temporal coherence, while the input is usually a sequence of desirable motions rather than a single image.

• •

  The CycleGAN could not provide similar poses compared with the ground truths. We can observe that almost all the poses were distorted, and the qualities of the corresponding images were constantly degraded.

We presented the visualization of the ablation study results on Subject1 in Figure 6. We observed that each component has different degrees of improvement to the results, including clear head, hand gestures, etc. Specifically:

• •

  The results produced by setting 1, which does not have the Keypoint Amplifier, have more noise than other settings, especially near the head area. Compared to the baseline setting 1, setting 2 with Keypoint Amplifier significantly improved the generated image qualities. For instance, the results of setting 1 in the first and second rows failed to synthesize complete head and hands, while the corresponding results of setting 2 fixed these errors.

• •

  As shown in the column of setting 7, DFC-Net with full components synthesized images with more accurate details and better qualities. In the second row of the results produced by setting 5, we can see the generated person lacked his arm, while the results of setting 7 with the augmented consistency loss from the support set can complete the arm part.