Delving into the Frequency: Temporally Consistent Human Motion Transfer in the Fourier Space

The workflow of a FFC block is presented in Fig. 3, which contains two main streams.The input $X$ is first divided into two parts. The top stream adopts traditional convolution to capture local information (i.e., $Y^{l}$). The bottom stream processes the feature in the frequency domain (i.e., Spectral Transform) to capture frequency information and global information (i.e., $Y^{g}$). Two streams complement each other by exchanging local and global information as follows:

In the Spectral Transform ($f_{g}$), as shown in the right figure of Fig. 3, the feature is processed sequentially with Real 2-D FFT, frequency domain convolution, and Inverse Real 2-D FFT.

As shown in the top of Fig. 2, we regularize the appearance in the frequency domain with FFC blocks. The Frequency-domain Appearance Regularization (FAR) is trained to reduce the noise and artifacts in the coarse frame to make the appearance consistent for adjacent frames. First, the coarse frames are downsampled with 3 FFC blocks. Then the frame features are processed with several FFC residual blocks in both pixel and frequency domain, aiming to reduce the artifacts and noise which are abnormal or discontinuous. At last, the final features are upsampled with 3 FFC blocks to reconstruct the frame.

To discover the gap between natural and synthetic videos, we analyze the temporal changes of adjacent frames in amplitude and phase separately, which is shown in the bottom of Fig. 2. We define the Temporal Frequency Change (TFC), i.e., the Temporal Amplitude Change (TAC) of $|F(u,v)|$ and Temporal Phase Change (TPC) of $\angle F(u,v)$ between adjacent frames $f_{t}$ and $f_{t-1}$ in a video, as

We analyzed the synthetic frames with low temporal consistency and drawn several observations, and several example heatmaps of TFC are shown in Fig. 4. First, for adjacent synthetic frames with low temporal consistency, TFCs scatter over various frequencies with larger values, while the ones of natural videos are mainly in the low-frequency components with smaller values. Second, the phase spectrum of natural videos contains more changes in the high-frequency components compared with the amplitude spectrum. This observation is consist with the conclusions in (Seshadrinathan and Bovik, 2010). The main content (i.e., intensity of the pixels) of adjacent frames does not change too much, which is reflected in the amplitude strum. The position of the pixels change but not very intensely, which is reflected in the phase spectrum.

Based on the above observation, we present how to utilize the findings for temporally consistent human motion transfer. The overview of the proposed framework is shown in bottom of Fig. 2. For any GAN-based human motion transfer model, we first calculate the temporal frequency changes (TFCs) between adjacent frames of referred videos and generated videos. Then a Weighted Temporal Frequency Regularization (WTFR) loss is obtained from the TFCs to guide the training of the model. By this means, the model can make the temporal patterns of generated videos approximate to natural videos in the frequency domain.

Given a source image $I^{S}$ and a frame sequence $\{f^{R}_{t}\}^{T}$ with the referenced motion, the model aims to synthesize a frame sequence $\{f^{S}_{t}\}^{T}$ conditioned on the appearance in $I^{S}$ and the motion in $\{f^{R}_{t}\}^{T}$, where $T$ is the length of the referenced sequence. Motions in adjacent frames exhibit temporal frequency changes which reflect different magnitudes in amplitude and phase spectra. To make the model learn the TFC of natural videos, we define the WTFR loss, which regularizes TAC and TPC of adjacent frames separately. For TAC, the loss function $L_{TAC}$ is formulated by

and the loss function $L_{TPC}$ for TPC is formulated by

where $w(u,v)=[(\frac{M}{2})^{2}+\frac{N}{2})^{2}]^{\delta}-[(u-\frac{M}{2})^{2}+(v-\frac{N}{2})^{2}]^{\delta}+1$ is a weighting term to penalize high-frequency components based on the observation that TFCs in natural videos are relatively small in high-frequency components as shown in Fig. 4). $\delta$ is a hyper-parameter to scale the magnitude.

Finally, the WTFR loss is denoted as

where $\alpha$ and $\beta$ are hyper-parameters. With the WTFR loss, the model can learn to approximate the frequency distribution of natural videos by minimizing the $L_{1}$ distance of TFCs between the referenced video and the synthetic video.

SoloDance (Wei et al., 2021b) We adopted C2F-FWN as our backbone, and thus we adopted the SoloDance dataset released by C2F-FWN for comparison. The SoloDance dataset contains 179 solo dance videos with 53,700 frames collected from the Internet. There are 143 human subjects with various clothes and performing complex solo dances in various backgrounds. This dataset is split into 153 and 26 videos for training and testing, respectively.

To validate the effectiveness of the FreMOTR, we compare the performance with the following HMT methods: FSV2V (Wang et al., 2019) that takes multi frames as input to improve temporal coherence, SGWGAN (Dong et al., 2018) and LWGAN (Liu et al., 2019b) that warp the features, ClothFlow (Han et al., 2019) that warp the image, and C2F-FWN (Wei et al., 2021b) that constrains temporal coherence by optical flow.

The C2F-FWN is the backbone in our experiments, but the authors did not release pretrained weights. Therefore, we trained the C2F-FWN based on the released code and settings, and report the reproduced performance. Performances of other methods are from  (Wei et al., 2021b).

We adopt the C2F-FWN that achieves the SOTA temporal consistency as the backbone for the SoloDance dataset. We adopt a three-stage training strategy. During the first stage, we train the backbone using the settings and hyper-parameters in the released codes of C2F-FWN until converge, then the parameters are fixed. During the second stage, the FAR is combined and trained with general image generation loss like adversarial loss (Goodfellow et al., 2014) and perceptual loss (Johnson et al., 2016) until converge. Finally, the WTFR is added to the general losses to finetune the FAR.

The experiments are conducted on four NVIDIA Tesla P40 GPUs with 24GB memory based on PyTorch (Adam et al., 2019). Following C2F-FWN, the input frames are resized and cropped to 192$\times$256. The Adam optimizer (Kingma and Ba, 2015) is used for optimization with a batch size of 8. The hyper-parameters $\alpha$, $\beta$, and $\delta$ for the WTFR loss in Eq. 6 are set to $0.5$, $1$, and $0.05$, respectively, to make the WTFR has the same order of magnitude of other losses. The FAR has 3 downsampling blocks, 3 upsampling blocks, and 18 residual blocks. The initialization of FAR is based on the pretrained FFC provided by Suvorov et al. (2022).

We also adopt only WTFR for ablation. After the first stage that the backbone is well trained, the WTFR is used to finetune the backbone until converge.

To measure the frame-level image quality, we adopt SSIM (Wang et al., 2004), PSNR, and LPIPS (Richard et al., 2018) following previous works (Wei et al., 2021b). Specifically, SSIM, PSNR, and LPIPS compare the pairwise similarity of synthetic frames and ground-truth frames. Higher value is better for SSIM and PSNR, while lower value is better for LPIPS. The frame-level metrics are averaged over all frames of all synthetic videos.

To evaluate the temporal consistency, we adopt Temporal Consistency Metric (TCM) (Yao et al., 2017). TCM is calculated from the warping error between frames. We first obtain the optical flow $o_{t}$ between adjacent referenced frames $f^{R}_{t}$ and $f^{R}_{t-1}$ using Flownet 2.0 (Ilg et al., 2017). The optical flow is used to obtain two adjacent frames by warping $f^{S}_{t-1}$ and $f^{R}_{t-1}$. Finally, TCM is obtained by the ratio of warping errors as

where $\operatorname{warp}(\cdot,\cdot)$ is the warping operation. The temporal metrics are averaged over all adjacent frames of all synthetic videos.

We present some of the synthesized frames in Fig. 5, from top to bottom are ground truth, results from C2F-FWN, and results from our FreMOTR. The results of C2F-FWN often contain artifacts, and the consistency of the appearance cannot be preserved even in three successive frames.

The visual results demonstrate that with the FFC that mitigates the incoherent appearance and the WTFR that closes the temporal frequency gap, FreMOTR significantly reduces the inconsistency between successive frames. Therefore, the FreMOTR can achieve a long-time consistency, and a sample is shown in Fig. 6. More results can be found in the appendix. The proposed FreMOTR eliminate irrational temporal inconsistency like the artifacts and noise, and thus the visual quality of individual frames is improved at the same time, which is consistent with the results in Table 1.

We combined the WTFR directly with the backbone to analyze the effectiveness of WTFR (i.e., only WTFR). We visualize the video-level TFC to obtain an observation of overall improvement. For a video with $\rm T$ frames, the mean TFC (i.e., $\overline{\rm TFC})$ for amplitude and phase over the entire video (i.e., $\overline{\rm TAC}$ and $\overline{\rm TPC}$) are formulated as

where $(u,v)$ is the coordinate on the spectra. Several samples are shown in Fig. 7. C2F-FWN considers temporal consistency based on optical flow, and the patterns of $\overline{\rm TFC}$ from C2F-FWN is similar with the ones of ground truth. However, the values are larger in general, especially in the low-frequency components. Our proposed WTFR can improve the synthetic results, making not only the distribution of $\overline{\rm TAC}$ and $\overline{\rm TPC}$ but also the values closer to natural videos.

The experimental results demonstrate that the proposed frequency-domain regularization (i.e., WTFR) indeed makes the temporal frequency changes of synthetic videos closer to the ones of natural videos, which component the pixel-domain supervision for human motion transfer. More comparisons of frames and spectra can be found in the appendix materials.

We first conducted the ablation study on the two components of FreMOTR. The results are presented in Table 1. “only FFC” refers to the model trained from the second stage without WTFR finetuning “only WTFR” refers to the backbone model finetuned with WTFR.

Specifically, the WTFR significantly improves the frame-level image quality, especially the LPIPS and PSNR. WTFR improves the TCM a lot at the same time. We analyze that WTFR reduce the irrational noise like artifacts, noise, and jitters that suddenly appear, and thus achieves better image quality and temporal consistency.

The FAR significantly improve the TCM. We infer that FAR mainly mitigate the incoherent appearance of the person and achieve much better temporal consistency. To make the entire image more natural, the FAR takes the entire image as input rather than only the body region, and the background may be slightly influenced, which lead to unsatisfied frame quality metrics. However, by observing the results, we did not observe image quality descending. In contrary, as shown in the qualitative analysis, although FreMOTR and “only FFC”” have similar or even worse image quality metrics with C2F-FWN, they have less obvious artifacts and unnatural patches.

For human motion transfer, both frame-level image quality and temporal consistency are important, and the Full FreMOTR achieves a well trade-off. The temporal consistency metric of FreMOTR is much better than the backbone and slightly worse than “only FFC”, while the image quality metrics like LPIPS and PSNR is better than “only FFC” and the backbone. It is reasonable that the smooth Furthermore, the qualitative analysis in Sec. 4.3 demonstrates that beyond the frame-level image quality, the visual performance is significantly improved. These results demonstrate that all the components of FreMOTR works, which benefit temporally consistent human motion transfer in different aspects.

WTFR is composed with amplitude and phase. To study the roles of each components, we combined the converged backbone (i.e., C2F-FWN (Wei et al., 2021b)) with several variation of WTFR, and the results are presented in Table 3. The WTFR is defined in Eq. 4, Eq. 5, and Eq. 6. In the table, “Phase” refers to phase only (i.e., $\alpha=0$ in Eq. 6), “Amplitude” refers to amplitude only (i.e., $\beta=0$ in Eq. 6). “TFR” means removing the weighted term $w$ (i.e., $w=1$ in Eq. 4 and Eq. 5). For the Mixed setting, only one of the components uses the weighted term. “AMP.+WP” means amplitude without the weighted term $w$ and phase with the weighted term $w$, while “PH.+WA” means phase without the weighted term $w$ and amplitude with the weighted term $w$.

First, the results demonstrate that all variations improve the temporal consistency, while the frame-level quality can also be improved. The amplitude part and phase part of WTFR are both effective. The TCM metrics of generated videos are improved by 11% $\sim$ 14% than the original models.

Second, the amplitude part is more important than the phase. As described in Sec. 3.1, amplitude reflects the intensity of the pixels while phase reflects the position. Therefore, minimizing amplitude fluctuation can guarantee the consistency of pixel intensity, which reduces irrational components that suddenly appear.

Third, the comparison between results of WTFR, TFR, and Mixed shows that the mixed variation of “PH. + WA” works best. The weighting term in WTFR is important for the model to pay more attention to high-frequency components, especially for the amplitude. This phenomenon is consistent with the observation in Fig. 4. The main content (i.e., intensity of the pixels) of adjacent frames does not change too much, and thus we should adopt larger penalty to the high-frequency changes for amplitude spectrum to make the synthetic frames more natural. On the other hand, the position of the pixels change but not very intensely, and thus we just adopt a uniform penalty for all components in the phase spectrum.