Learning Temporally and Semantically Consistent Unpaired Video-to-video Translation Through Pseudo-Supervision From Synthetic Optical Flow

Inspired by recent progress in generative adversarial networks, several early image-to-image translation methods (Denton et al. 2015; Radford, Metz, and Chintala 2016; Zhu et al. 2016; Zhao, Mathieu, and LeCun 2017; Isola et al. 2017; Shetty, Fritz, and Schiele 2018; Choi et al. 2018; Zhu et al. 2017b) have been proposed to tackle problems in various applications such as image generating and editing. However, all these methods require costly input-output image pairs. Many methods (Liu and Tuzel 2016; Taigman, Polyak, and Wolf 2017; Shrivastava et al. 2017) have also been proposed to address the unpaired version of the problem. Independently, cycle-consistency constraint has been introduced (Zhu et al. 2017a; Liu, Breuel, and Kautz 2017; Kim et al. 2017) that significantly improves the quality of translation by forcing cyclic consistency of an image before and after translations and inspires many following works (Yi et al. 2017; Hoffman et al. 2018; Huang et al. 2018; Yang, Xie, and Wang 2018; Liu et al. 2019; Baek et al. 2020).

On the basis of image-to-image translation, the video-to-video translation problem is firstly explored in vid2vid (Wang et al. 2018). The quality of synthesized videos is further optimized in (Wei et al. 2018; Mallya et al. 2020) by incorporating long-term information. Few-shot vid2vid (Wang et al. 2019) alleviates the dependency of the model on a huge dataset and enhances the generalization capability of the model. Similar to its image counterpart, these methods also require paired annotated videos, which is expensive to acquire. Therefore, to address unpaired video-to-video translation, a spatiotemporal 3D translator is proposed in (Bashkirova, Usman, and Saenko 2018) on the basis of existing image-to-image translation networks. To regularize spatiotemporal consistency, Recycle-GAN (Bansal et al. 2018) integrates temporal factors into its Cycle-GAN-based model by predicting future frames. Mocycle-GAN (Chen et al. 2019) and STC-V2V (Park et al. 2019) adopt optical flow to improve the accuracy of their spatiotemporal consistency constraints. To further improve the coherence of the generated video, a spatiotemporal discriminator unit, as well as a self-supervised ping-pong loss, is proposed in TecoGAN (Chu et al. 2020) for long-term temporal consistency regularization. UVIT (Liu et al. 2021) applies an RNN-based translator as well as a self-supervised video interpolation loss to incorporate temporal information and generates videos in a recurrent manner. Another branch of works (Bonneel et al. 2015; Lai et al. 2018; Lei, Xing, and Chen 2020) seek to improve the temporal consistency of videos in a post-processing manner that is independent from the translating process, which can limit the quality of the improvement.

Although spatiotemporal consistency constraints in these methods improve the quality of the generated videos, these constraints rely on some form of motion estimation that is not perfect. As a result, errors in motion estimation undermine the spatiotemporal consistency constraints and thereby deteriorates the quality of the generated videos. To investigate the effect of the accuracy in the estimation of motions on the quality of spatiotemporal consistency constraint, we conducted a pilot experiment on STC-V2V (Park et al. 2019) with different optical flow estimating methods and calculate the warping error. The definition of the warping error will be discussed in later sections. Besides the original method that adopts Flownet2 (Ilg et al. 2017) to estimate flow between two temporally adjacent frames, we also ran an experiment using the same model but with intentionally reversed order of the input frames to simulate a less accurate estimation, and another experiment with a more advanced flow estimator RAFT (Teed and Deng 2020).

As presented in the results in Table 1, the improvement of the optical flow estimating method contributes to the decrease in the warping error, which indicates the enhancement of the temporal stability of the resulting videos. Therefore, we conclude that regularizing the spatiotemporal consistency with the absence of accurate motion estimation limits the performance of the generator to conduct stable video translation.

Compared with existing unpaired video-to-video translation methods, our work focuses on providing more accurate and efficient spatiotemporal consistency regularization through the pseudo-supervision from synthetic optical flow instead of estimated motions between frames.

Different from unpaired video-to-video translation, video style transfer methods are mainly built on the basis of existing non-adversarial image style transfer and focus on the improvement of temporal consistency of the resulting videos given content videos and style images. Efficient feed-forward networks for fast video style transfer with different variants of temporal constraints (Ruder, Dosovitskiy, and Brox 2016; Huang et al. 2017; Gupta et al. 2017; Chen et al. 2017; Gao et al. 2018; Chen et al. 2020) are adopted to optimize temporal consistency. However, all these methods either require ground truth optical flow from the dataset or suffer from the incorrect estimation of optical flow, which limits their performance. To address these problems, CompoundVST (Wang et al. 2020) proposes a single-frame based video style transfer method that adopts compound regularization to learns temporal consistency through synthetic optical flow while LinearVST (Li et al. 2019) and MCCNet (Deng et al. 2021) seek to enhance the learning of arbitrary style transfer by modifying the architecture of models.

Noticing the drawbacks of existing spatiotemporal constraints and inspired by the recent progress in CompoundVST (Wang et al. 2020), we integrate the pseudo-supervision from synthetic optical flow into our Cycle-GAN-based paradigm. Note that, compared to CompoundVST, our work focuses on the cross-domain unpaired video-to-video translation problem and not just video style transfer. Also, we propose a novel regularization paradigm that more effectively enforces spatiotemporal consistency in both domains and therefore facilitates not only temporally stable but also semantically consistent translation. Recent methods (Eilertsen, Mantiuk, and Unger 2019; Zhang et al. 2021) have also focused on enhancing video quality in different specific tasks through the regularization of consistency across frames based on motion. Nevertheless, we note that our method addresses the more general problem of unpaired video-to-video translation.

To translate videos to the other domain, we train the generators and discriminators adversarially in a zero-sum game as in (Goodfellow et al. 2014). Practically, we optimize our discriminators $D_{X}/D_{Y}$ so that they can correctly distinguish synthetic images $\hat{x}_{t}/\hat{y}_{s}$ from real images $x_{t}/y_{s}$. At the same time, generators are trained to synthesize realistic images that can fool the discriminator. Therefore, the adversarial loss $\mathcal{L}_{adv}$ is given by

Given a source frame $x_{t}$ as the $t$-th frame from source video $x_{1:T}$, we simulate the synthetic future frame $\tilde{x}_{t+1}$ by warping the current frame with a synthetic optical flow $\tilde{f}$ as $\tilde{x}_{t+1}=W(x_{t},\tilde{f})=F(x_{t},\tilde{f})+\Delta$ as in CompoundVST (Wang et al. 2020). Here $F$ represents the warping operation with the synthetic optical flow and $\Delta$ represents pixel-level random Gaussian noise. Refer to Appendix A for more details regarding the generation of the synthetic optical flow.

Unsupervised Recycle Loss: Given an original frame $x_{t}/y_{s}$ and its translated frame $\hat{x}_{t}/\hat{y}_{s}$, we simulate their synthetic following frame $\tilde{x}_{t+1}/\tilde{y}_{s+1}$ and $W(\hat{x}_{t},\tilde{f})/W(\hat{y}_{s},\tilde{f})$ through a warping operation with an identical synthetic optical flow $\tilde{f}$. After that, we map $W(\hat{x}_{t},\tilde{f})/W(\hat{y}_{s},\tilde{f})$ back to the original domain as in Recycle-GAN and enforce its consistency with $\tilde{x}_{t+1}/\tilde{y}_{s+1}$ using unsupervised recycle loss $\mathcal{L}_{ur}$ given by

Compared with Recycle-GAN (Bansal et al. 2018) that applies a pixel-level frame predictor to estimate the future frame, our paradigm uses the synthetic optical flow that ensures the consistency of the motion across domains. Consequently, the error from motion estimation becomes irrelevant to the translation process resulting in guaranteed accuracy of spatiotemporal consistency guidance.

Unsupervised Spatial Loss: In parallel with $\mathcal{L}_{ur}$, our method also applies an unsupervised spatial loss $\mathcal{L}_{us}$ that enforces the consistency between $\tilde{x}_{t+1}/\tilde{y}_{s+1}$ and $W(\hat{x}_{t},\tilde{f})/$W($\hat{y}_{s},\tilde{f}$) in the opposite direction(see Equation (3)). Practically, $\tilde{x}_{t+1}/\tilde{y}_{s+1}$ is mapped to the other domain to be consistent with $W(\hat{x}_{t},\tilde{f})/$W($\hat{y}_{s},\tilde{f}$) so that the warping and the translation is commutative to each other.

Guided by the pseudo-supervision from synthetic optical flow, our unsupervised losses include errors only from inconsistent translation processes in $G_{X}$ and $G_{Y}$, and minimizing the loss value leads to direct improvement in the quality of translation without misleading the generators.

Combining all loss functions we listed above, we present the full objective of our paradigm to optimize a single-frame based unpaired video-to-video translation system as

The weight of our unsupervised losses $\lambda_{ur}$ and $\lambda_{us}$ are set to 10 for all experiments. Note that additional losses can be added to the objective as needed for the translation task.

In this experiment, we validate the temporal consistency and visual quality of the translated videos from our method. Cycle-GAN (Zhu et al. 2017a), Recycle-GAN (Bansal et al. 2018), and STC-V2V (Park et al. 2019) are included as existing state-of-the-art methods for comparison. Additionally, we adopt the content loss from STC-V2V during the training of our method and set $\lambda_{cont}$ as $1$ to constraint the similarity of content in images during translation. Qualitative results are presented in Figure 4.

To further verify that our method performs semantically consistent video-to-video translation, we conducted the video-to-labels and labels-to-video experiments following protocols in previous works (Bansal et al. 2018; Chen et al. 2019; Liu et al. 2021). In this scenario, we translate the videos to their corresponding pixel-level semantic segmentation labels in Viper on the basis of the architecture of Cycle-GAN (Zhu et al. 2017a) and MUNIT (Huang et al. 2018). Besides our unsupervised losses, we also applied the cycle loss following (Zhu et al. 2017a; Bansal et al. 2018) and set $\lambda_{cyc}$ as $10$ during our training process to further enhance the regularization for the sake of fair comparison. The quality of the translation, in terms of semantic consistency, was evaluated quantitatively in both directions. We incorporated Cycle-GAN (Zhu et al. 2017a), Recycle-GAN (Bansal et al. 2018), and STC-V2V (Park et al. 2019) as baseline methods in the following experiments and compare their performance with ours. Additionally, we suppressed the regularization for the video domain in our paradigm to avoid generating vague contents from the regions in the label domain that have the same semantic labels. Further discussions for this setting are available in ablation studies and Appendix A. Qualitative results are shown in Figure 5.

To verify the effectiveness of our unsupervised spatiotemporal consistency constraints and other relevant components, we ran ablation studies on the video-to-labels translation task. More ablation studies conducted on Viper-to-Cityscapes as well as to explore the effect of the synthetic optical flow are available in Appendix B.

For video-to-labels translation, we ran experiments with each of our loss functions and their meaningful combinations. Additionally, we also turned off the video domain regularization suppression to investigate its effects in video generation. The performance of these experiments was evaluated in both segmentation scores for the video-to-labels translation and FCN-scores for the labels-to-video translation. Based on the results shown in Table 8, we can observe that each of our unsupervised losses cannot achieve semantically consistent translation independently, while their combination performs significantly better in the video-to-labels translation. On that basis, enabling video domain regularization suppression further improves the quality of the videos translated from labels, as it prevents local details suppression. Finally, the application of the cycle loss contributes another leap in the quality of the generated videos, showing the effectiveness of the cyclic spatial constraints in preserving semantic consistency. Qualitative results are presented in Appendix C for a more intuitive understanding.

To verify the effectiveness of using the proposed method to generate synthetic optical flow, we compare three optical flow generation methods with our paradigm: 1) optical flow estimated through Flownet2, 2) optical flow synthesized with only translation motions, and 3) optical flow synthesized with only scaling motions. We compare the warping error on the Viper-to-Cityscapes translation experiment. As presented in Table 10, estimated optical flow deteriorates the translation possibly due to artifacts of discontinuity as compared to our smooth synthetic flow. The artifacts combined with our strong consistency constraints undermines adverserial training and thereby the quality of translation. On the other hand, optical flow with only low-level motions, such as translation or scaling, cannot provide rich information for temporal and semantic consistency for the training process. In contrast, our synthetic flow ensures the quality of the optical flow with no flow discontinuity, providing stronger support to our effective regularization paradigm.

Finally, we tested hyper-parameter sensitivity in the Viper-to-Cityscapes translation with $(\lambda_{UR},\lambda_{US})$ as $(1,10)$, $(5,10)$, $(10,1)$, and $(10,5)$. The resulting warping error are presented in Table 9. The results show that when either of the weights is not large enough, the effectiveness of the consistency constraint will be compromised. On the other hand, when the ratio of the weights gets closer to 1, our method yields stable results, revealing the robustness of our method against hyper-parameters.