Sketch Me A Video

We have reviewed existing works related to video creation task, and find that this is the first work to do sketch-to-video generation from only two given rough bad-drwan sketches. Nevertheless, we still present three kinds of video generation works which are mostly related to ours, as described before, which are divided into video translation  (Wang et al., [n.d.]) (Wang, 2019) (Chen et al., 2019) and prediction (Wang et al., [n.d.]) (Wang, 2019), video interpolation (Shen et al., 2021; Yan et al., 2021; Park et al., 2021; Bao et al., 2021), and video inbetweening  (Wang et al., 2019) (Xiaoyu et al., 2020). These works have been analyzed in the previous section and are all different from our work which rely on rough sketches only. We next will give some review on sketch processing.

Free-hand sketches are highly illustrative, and have been widely used by humans to depict objects or stories for a long time. The recent popularization of touchscreen devices has made sketch creation a much easier task than ever and consequently made sketch-oriented applications increasingly popular. The progress of deep learning has immensely benefited free-hand sketch research and applications. However the sketches could be divided into two categories. The first one takes well-drawn sketch which is usually got via edge detection from drawings or from artists, which has stable distribution and could be easily translated. The second category is free sketch drawing (Skecthy GAN (Chen and Hays, 2018), Sketh me that shoe (Yu et al., 2016)), whose distribution is too causal and free-form, and lacks structure mapping between the sketch and object in real world. That makes it hard to expolre. Benefiting from the development of feature engineering, the early works use image retreive method to find a closest sample in semantic space. Direcly translating those sketches into multimedia domain, could cause distortion. Therefore, temporal information is often introduced to help synthesize the video.

Most video creation works  (Wang et al., [n.d.]) (Wang, 2019) (Chen et al., 2019) do video translation based on conditional input sequences, which contain the guiding temporal information serving as auxiliary input to help synthesize the target video. For example, video interpolation (Wang et al., 2018) (Jiang et al., 2018) as a classical vision problem aiming to synthesize high FPS videos from a low one, can synthesize serval more frames from given two based on given temporal reference. The SuperSlomo (Jiang et al., 2018) as a famous video interpolation method which has been widely used in video creation, also leverages on extra guide input to interpolate more frames.

When no conditional input is available, some works (Wang et al., 2019) try to synthesize the conditional sequences as auxiliary input by employing prior knowledge such as object normal behaviours, daily activities, etc. There also exist some works (Xiaoyu et al., 2020)  (Xu, 2019) which even directly calculate the motion changes based on certain mathematical distribution assumptions, and leverage on the derived motions to help synthesize the video. Our work on the contrary does not utilize any temporal sequence or any other form of guide information, and purely use the rough sketches to create the video.

Our goal is to generate a smooth video from the given two bad-drawn face sketches in an interactive manner. The video should have the natural motion and meanwhile the attributes of the sketch such as identity, texture, etc.

Our Pipeline. We divide this challenging task into two stages:

• •

  Stage-I model: It takes the two bad-drawn sketches to generate two reasonable start & end frames, and meanwhile produce the intermediate semantic feature sequences to be used in Stage-II.

• •

  Stage-II model: It takes semantic feature sequences from the Stage-I model and blends them hierarchically with a motion variable extracted from the TMP module (in training phase) or the standard normal distribution (in the testing phase) to synthesize the final video.

Symbols Definition. We denote the training video and the output video as $v$ and $\hat{v}$, respectively. The frame number of the two videos is $N$, and each frame in $\hat{v}$ is denoted by $\hat{v}_{o},o\in[1,N]$. The start sketch and end sketch are denoted by $s_{s}$ and $s_{e}$, respectively.

In Stage-I, we train an FRP module that takes input facial sketches to generate realistic frames and hierarchical semantic feature sequences, as shown in Fig.2.

Because the input of our Stage-I model is bad-drawn sketches, the developed FRP module is expected to tackle the out-of-domain challenge mentioned before. We train several partial encoders denoted by $E_{part}$ to project the sketch into semantic manifolds. The $E_{part}$ embeds different sketch parts into different embeddings seperately, which renders the model to be able to disentangle various facial attributes such as mouth, eye, nose, for subsequent feature retrieve and projection.

To retrieve for the input sketch part feature, we need to build a codebase consisting of rich semantic embeddings of various facial skech parts by using the $E_{part}$ to encode the whole dataset. Then, for each input partial sketch, we can use the K-nearest neighbor algorithm to retrieve the top K closest embeddings which may represent the out-of-domain sketches in the semantic space of facial parts sketches. We follow the locally linear embedding (LLE) (Roweis and Saul, 2000) algorithm to project the Top-K samples into the embeddings of the out-of-domain sketch.

After getting the partial sketch embeddings, we still need to design a module to get high-quality images (e.g., start and end) and meanwhile map those partial sketch embeddings into the semantic space of natural images. Hence, we adopt the widely used Pix2PixHD (Wang et al., 2018) as a backbone to do this task. However, we observe that directly decoding these partial embeddings into a realistic image faces the challenge that features of different facial parts are heavily entangled, which causes the difficulty of establishing a spcified atttribute manipulation mapping between input and output. To alleviate this, we exploit several partial decoders $D_{part}$ to reassemble the retrieved embeddings into an image-like feature pool, where each feature in the image corresponds to its original facial part (attribute) location in the input sketch. We then feed them into the Pix2PixHD  backbone. Besides, we apply a feature pyramid block on the decoder of Pix2PixHD (Wang et al., 2018) backbone, which provides hierarchy information of different scales of the realistic face to benefit the video generation process in Stage-II.

We denote the whole process of Stage-I as:

where the $G_{frp}$ denotes the developed FRP module, and $f$ denotes a set of hierarchy features output by the feature pyramid block. Note that the number of features in a $f$ is equal to the number of the upsampling blocks. The subscript $s$ and $e$ denote the start and end frames, respectively.

In Stage-II, we propose a model that generates a video with only two bad-drawn sketches. The Temporal Motion Projection (TMP) module, used in training phase only, aims to project a training video sequence into a motion space that will be used in test phase. Then, the developed Feature Blending (FB) module fuses the motion variable from TMP and the semantic features from Stage-I in a disentangled way. The FB module generates a merging mask with bidirectional optical flows for both the start and end frames. Finally, the optical flow merging (OFM) module synthesizes the final video. Additionally, A smooth motion loss is designed to constrain the motion space to have only our desired directions by removing those unpredictable directions.

Temporal Motion Projection Module. With the semantic features synthesized in Stage-I as the auxiliary, the challenge now is the lack of guiding temporal information between the sketches. We use the TMP module to project the training video into a standard normal distribution in training phase, while replacing the motion variable with a noise $z$ during the test phase to tackle this challenge. Leveraging the idea of cVAE (Xue, 2016), we project the training video into a motion space. Specifically, we use multiple-layer perceptrons (MLP) to map the embeddings of TMP into mean and log variance to calculate the motion variable. We use the Kullback-Leibler (KL) divergence as a loss function to make the distribution of the motion variables (referred to as motion space) conform to the standard distribution.

As the motion space from the training video may present weak relevance with the input sketches, the TMP module thus needs to consider encoding the motion between the input sketches to correlate the video motion with sketches. Besides, during the Stage-II training stage, the input sketches are directly from the start & last frame edges of the training video v, and the training sketches are well generated through the edge detection method (HED (Xie and Tu, 2017)) with an edge simplification method(SimpleEdge (Simo-Serra et al., 2016)). We thus regard the sketch as a solid hint to the motion modelling, which benefits the optical flow generation. Therefore, we concatenate v with $s_{s}$, $s_{e}$ as the input of the TMP module. This process could be denoted as:

Here we simplify the TMP module as an encoder $E_{TMP}$, and the $f_{m}$ is the motion variable. During the test, $f_{m}$ is sampled from standard distribution $N(0,1)$ and denoted as $z$.

Feature Blending Module. After sampling a motion variable from the motion space generated by TMP, we need to further design afeature blending (FB) module to blend the motion variable $f_{m}$ with the extracted semantic features ($f_{s}$, $f_{e}$) from Stage-I via a video decoder. We concatenate the first item in $f_{s}$, $f_{m}$, and the first item in $f_{e}$ and feed the fused feature into the video decoder. To complement the large scale information of targets, the upsampling block of the decoder also takes features from $f_{s}$ and $f_{m}$ at the same scale.

The output of the FB module is optical flows with masks and a visible mask. The two pairs of optical flows for the start and end frames are bidirectional, where the reverse direction is to check the forward-backward consistency from the training video. The visible mask $m_{v}$ is a soft mask to coordinate the visibility of optical flows from the start and end frames. This process could be denoted as:

where the $o$ is an optical flow sequence, and the $m$ is a mask sequence. The subscript $s$ and $e$ denote the flow is from the start and end frame, respectively. The subscript $f$ and $b$ represent the forward and backward flow, respectively.

Optical Flow Merging Module. The optical flow merging (OFM) module generates the final video from the outputs of the FB module. To manage the flows from both the start and end frame and make the $\hat{v}$ stable, the OFM module merges the above optical flows and mask with following operation:

where the $W(\cdot,\cdot)$ is the warping process, and the $\odot$ denotes Hadamard product.

Although we can generate a natural motion space via the TMP , the motion in the video dataset may be a random and free-form action beyond control. For example, in Fig.1, the input sketches are a man with a neutral mouth and a big smile mouth, respectively, but the corresponding motion in the training video could be started from a neutral mouth, then to blinking or talking for a whole, which does not conform to that of the bad-drawn sketches. In fact, the ideal motion is expected to present a gradual change mode from the start to the end sketch, in our bad-drawn Sketch-to-Video generation task. We have to find a method to control the generated motion to present similar distribution with the input sketches changes.

An intuitive way is to directly introduce a constraint to the synthesized results, but this could cause the output to have a big gap with the ground truth, making the model hard to converge. To work on that, we carefully design a smooth motion loss to the optical flows $o_{ef}$ and $o_{sf}$ to make the motion grow gradually without abrupt changes. The $\mathcal{L}_{sm}$ is formed as:

Here, the $o_{xf}[\cdot]$ is the index of the forward optical flow sequence $o_{xf}$, and the $F_{li}$ denotes the linear interlpolation. The ablation study 4.4 in the experiment part well validates that after applying such a loss to the model, the unwanted motion disappeared and the motion change becomes smooth meanwhile keeping natural.

On the other hand, we consider that because the ground truth is not directly but through the forward-backward consistency to penalize the forward optical flow, this makes the backward flow ($o_{eb}$ and $o_{sb}$) serve as an effective intermidate to further ensure the smoothness of generated motions.

We firstly train the Stage-I model, then using the trained Stage-I model to train Stage-II. The Stage-I model uses the same loss function as pix2pixHD , except introducing a pixel loss between the sketches of synthesized images and the training sketches. Besides the $\mathcal{L}_{sm}$, we also propose several other losses to train the stage-II model as follows.

A pixel-level loss is used between the input $v$ and output video $\hat{v}$ to reconstruct the training video, formulated as:

Following [As-rigid-aspossible stereo under second order smoothness priors., an unbiased second-order prior for high-accuracy motion estimation], a smoothness constraint loss $\mathcal{L}_{osc}$ encourages the flow to be similar in a local neighborhood as follows,

As mentioned before, we seperately apply a bidirectional frame reconstruction loss $\mathcal{L}_{bfr}$ to the start & end frame. For the start frame, the bidirectional frame reconstruction loss $\mathcal{L}_{sbfr}$ can be formulated as

The end frame bidirectional reconstruction loss can be formulated as $\mathcal{L}_{ebfr}$:

The forward-backward consistency loss $\mathcal{L}_{fbc}$ for unmasked regions can be formulated as:

As there is no standard sketch-to-video benchmark for use, we thus jointly utilize the public CelebAMask-HQ (Lee et al., 2020) dataset and VoxCeleb2 (Chung et al., 2018) dataset to conduct experiments and evaluate our proposed method on the sketch-to-video task. We train our Stage-1 model on the CelebAMask-HQ (Lee et al., 2020) dataset to perform realisitic generation and train our Stage-2 model on the VoxCeleb2 (Chung et al., 2018) dataset to better create temporal information for video synthesis. CelebAMask-HQ is a large-scale face image dataset with 30,000 high-resolution face images. We randomly select $30\%$ for testing and the rest for training. VoxCeleb2 is a famous dataset consisting of talking head videos (224p videos at 25fps) which has 1 million utterances for 6,112 celebrities. We use 145,569 videos for train and 4,911 for testing. Note that our validation samples in qualitative experiments are drawn by human artists.

We get the sketch image by firstly performing edge detection via HED (Xie and Tu, 2017). Then, the images in the CelebAMask-HQ dataset are further simplified by  (Simo-Serra et al., 2018) to mimic human drawing. Note that we only need the sketches of the first and last frame in videos and abandon others as input. The images and videos are rescaled to $256\times 256$ pixels in all experimental settings.

We ask the participates to choose the video which one makes them feel more natural and gradual. We ask 10 particaptes to review each pair.

In the qualitative experiments, we propose several baselines, and conduct carefully designed experiments. We compare the performance and find out the real contribution of each stage of our model in this subsection. (Please zoom in to see the differences.)

Baselines As this is the first work to create videos from given two rough hand-drawn sketches, we propose several baselines to compare current works with ours.

• •

  P2P+SS (Wang et al., 2018) (Jiang et al., 2018). Video interpolation is a classical vision problem aiming to synthesize high FPS videos from a low one, some of them can synthesize serval more frames from given two. The SuperSlomo (Jiang et al., 2018) (SS) is a famous video interpolation method which has been widely used in video creation. Meanwhile, the Pix2PixHD (Wang et al., 2018) (P2P) is one of the most famous image translation networks. We combine these two works to build a pipeline, in which the P2P method translates the input sketches into images, and the SS method caculates a linear flow between the two frames as input.

• •

  P2P+S2 (Wang et al., 2018). To fairly compare the P2P+SS baseline with ours and in case the combination in P2P+SS makes the other slow down, we replace our Stage-I method with Pix2PixHD and keep our Stage-II (S2) method unchanged, which makes the comparison between S2 and SS a controlled experiment.

• •

  S1+SS (Jiang et al., 2018). For a more rigorous comparison of controlled experiments and compare the P2P with our the Stage-I model. In this setting, we combine our Stage-I method and SuperSlomo to find out the ability of video creation. Note the we modified the

Analysis of Qualitative Results. As shown in Fig.3, all baselines with the P2P (Wang et al., 2018) as stage-I method produce a seriously distorted face in the generated videos. As we analyzed in Section.3.2, due to the out-of-domain problem of rough hand-drawn sketches, the image translation model P2P (Wang et al., 2018) takes a sketch that has a free-form and different distribution not covered in the training datasets. This out-of-domain problem makes the result of the P2P seriously distorted. However, we find out all baselines with our Stage-I(S1) models have excellent and consistent semantics. We believe our Stage-I method alleviates the out-of-domain problem to a certain extent.

As to the terms of Stage-II, we mainly pay attention to the video generation performance. The generated qualities are found to depend on the choice of the Stage-I method. When stage-I is P2P, both the P2P+SS and the P2P+S2 are not good. The start and end frames are distorted and have a different identity. The results of both methods are gradually changing. We can see the P2P+SS baseline has a linear-like change, and the middle frames are blurry. However, one with our S2 module gradually changes the identity from the start to the end frame, and we can still see the human face organs of each frame, especially the eyes, which are lost in the P2P+SS baseline. In the S1+SS and our model(S1+S2), the difference is the teeth are buried in S1+SS(Please Zoom in to check the details.), where the motion is not linear to the underlying assumption of SS. Based on the above observation, we believe our stage-II method (S2) has better motion generation performance than the SS, and it could synthesize natural and gradual motion and has a learned prior.

We iteratively remove the smooth motion loss and visible mask to find out the actual function of them and whether these components work as expected or not. As shown in Fig.4, the generated video of ours gradually and naturally changes from a big smile to pursed lips expression. However, without the smooth motion loss, the motion would be random and the big smile firstly suddenly becomes a snickering expression from the third to the fifth frame, then becomes a pursed lips expression. As we mentioned in Section.3.4, this motion may be reasonable in the training dataset, but it makes the motion beyond the control of the given two sketches. As that hardly appears in the whole model, we believe that indicates the smooth motion loss works as expected.

To simulate the daily application scenario of video creation, we evaluate the interactive video synthesis ability of the model by conducting experiments as follows. Firstly the user uses a sketching input as an initial sketch and then interactively changes various parts of the sketch, e.g., emotion, haircut, and facial organs. Our model synthesizes the video between every two sketches.

The first coordinate system shows a casual way of face editing, and it shows the robustness of our method that can handle the regular video generation by changing expressions. The second coordinate system test some extreme expression changes, e.g., from the big smile face to the very sad face. The last two coordinate systems test entertaining video generation processes like growing or fading a beard or pout face. These interaction test results prove that our model has the potential ability to create video for normal users in practical use.