StyleCrafter: Taming Stylized Video Diffusion with Reference-Augmented Adapter Learning

We use the stylistic images from two publicly available datasets, i.e. WikiArt (Phillips and Mackintosh, 2011) and a subset of Laion-Aesthetics (Schuhmann et al., 2022) (aesthetics score above 6.5). In the original image-caption pairs, we observe that the captions generally contain both content and style descriptions, and some of them do not match the image content well. To promote the content-style decoupling, we use BLIP-2 (Li et al., 2023a) to regenerate captions for the images and remove certain forms of style description (e.g., a painting of) with regular expressions. In addition, as an image contains both style and content information, it is necessary to construct a decoupling supervision strategy to guarantee the extracted style feature free of content features. Although a stylistic image may contain different local style patterns (Park and Lee, 2019; Huo et al., 2021; Chen et al., 2023b), we regard that a large crop of an image(e.g. 50% of the image) still preserves a similar style representation with the full image. Based on this insight, we process each stylistic image to obtain the target image and style image through different strategies: for target image, we scale the shorter side of the image to 512 and then crop the target content from the central area; for style image, we scale the shorter side of the image to 800 and randomly crop a local patch with $512\times 512$. This approach reduces the overlap between the style reference and generation target, while still preserving the global style semantics complete and consistent.

CLIP (Radford et al., 2021) has demonstrated remarkable capability in extracting visual features from open-domain images. To capitalize on this advantage, we employ a pre-trained CLIP image encoder as a feature extractor. Specifically, we utilize both the global semantic token and the full $256$ local tokens (i.e., from the final layer of the Transformer) since our desired style embedding should not only serve as an accurate style trigger for the T2V model, but also provide auxiliary feature references. As image tokens encompass both style and content information, we further employ a trainable Query Transformer (Q-Former) (Li et al., 2023a) to extract style embedding $\mathbf{F}_{s}$. We create $N$ learnable style query embeddings as input for the Q-Former, which interact with image features through self-attention layers. Note that this is a commonly adopted architecture for visual condition extraction (Li et al., 2023a; Shi et al., 2023; Ye et al., 2023; Xing et al., 2023). But it is the style-content fusion mechanism that makes our proposed design novel and insightful for style modulation, as detailed below.

With the extracted style embedding, there are two ways to combine the style and text conditions, including (i) attach-to-text (Huang et al., 2023a; Li et al., 2023b; Ramesh et al., 2022): attach the style embedding to the text embedding and then interact with the backbone feature via the originally text-based cross-attention as a whole; (ii) dual cross-attention (Wei et al., 2023; Ye et al., 2023): adding a new cross-attention module for the style embedding and then fuse the text-conditioned feature and style-conditioned feature. According to our experiment (see Sec. 4.4), solution (ii) surpasses solution (i) in disentangling the roles of text and style conditions, therefore we have adopted it as our final solution. The formula can be written as:

where $\mathbf{F}_{in}^{i}$ denotes the backbone feature of layer $i$, LN denotes layer normalization, and TCA and SCA denote text-based cross attention and style-based cross attention respectively. $s^{i}$ is a scale factor learned by a context-aware scale factor prediction network, to balance the magnitudes of text-based feature and style-based feature. The motivation is that different stylistic genres may have different emphasis on content expression. For example, the abstract styles tend to diminish the concreteness of the content, while realism styles tend to highlight the accuracy and specificity of the content. So, we propose a context-aware scale factor prediction network to predict fusion scale factors according to the input contexts. Specifically, we create a learnable factor query, it interacts with textual features $\mathbf{F}_{t}$ and style features $\mathbf{F}_{s}$ to generate scale features via a Q-Former and then project it into layer-wise scale factors $\mathbf{s}\in\mathbb{R}^{16}$. Figure 3 illustrates the learned scale factors across multiple contexts. It shows that the adaptive scale factors have a strong correlation with style genres while also depending on the text prompts. Style references with rich style-semantics(i.e., ukiyo-e style) typically yield higher scale factors to emphasize style; while complex prompts tend to produce lower scale factors to enhance content control. This is consistent with our hypothesis to motivate our design.

Unlike T2I models, video models exhibit a higher sensitivity to style guidance due to their limited stylized generation capabilities. Using a unified $\lambda$ for both style and context guidance may lead to undesirable generation results. Regarding this, we adopt a more flexible mechanism for multiple conditions classifier-free guidance. Building upon the vanilla text-guided classifier-free guidance, which controls context alignment by contrasting textual-conditioned distribution $\epsilon(z_{t},c_{t})$ with unconditional distribution $\epsilon(z_{t},\varnothing)$, we introduce the style guidance with $\lambda_{s}$ by emphasizing the difference between the text-style-guided distribution $\epsilon(z_{t},c_{t},c_{s})$ and the text-guided distribution $\epsilon(z_{t},c_{t})$. The complete formulation is as below:

where $c_{t}$ and $c_{s}$ denote textual and style condition respectively. $\varnothing$ denotes using no text or style conditions. In our experiment, we follow the recommended configuration of text guidance in VideoCrafter (Chen et al., 2023a), setting $\lambda_{t}=15.0$, while the style guidance is configured with $\lambda_{s}=7.5$ empirically. Similarly, we set $\lambda_{t}=7.5$ and $\lambda_{s}=5.0$ for style-guided image generation.

We adopt the VideoCrafter (Chen et al., 2023a) as our base T2V model, which shares the same spatial weights with Stable Diffusion 2.1. We first train the style modulation on image dataset, i.e. WikiArt (Phillips and Mackintosh, 2011) and Laion-Aesthetics-6.5+ (Schuhmann et al., 2022) for 40k steps with a batch size of 32 per GPU. In the second stage, we froze the style modulation part and only train temporal blocks of VideoCrafter, we jointly train image datasets and video datasets(subset of WebVid-10M (Bain et al., 2021)) for 20k steps with a batch size of 1 on video data and 16 on image data, sampling image batches with a ratio of 20%. The training process is performed on 8 A100 GPUs and can be completed within 3 days. Furthermore, to ensure a fair comparison with some SDXL-based models (Ye et al., 2023; Hertz et al., 2023) on stylized image generation, we also trained the first stage of StyleCrafter on SDXL (Podell et al., 2023a).

To evaluate the effectiveness and generalizability of our method, we construct testsets comprising content prompts and style references. For content prompts, we use GPT-4 (OpenAI, 2023) to generate recognizable textual descriptions from four meta-categories (human, animal, object, and landscape). We manually filter out low-quality prompts, retaining 20 image prompts and 12 video prompts. For style references, we collect 20 stylized images and 8 sets of style images with multi-reference (each contains 5 to 7 images in similar styles) from the Internet. In total, the test set contains 400 pairs for stylized image generation, and 300 pairs for stylized video generation (240 single-reference pairs and 60 multi-reference pairs). Details are available in the supplementary materials.

Following previous practice (Zhang et al., 2023; Sohn et al., 2023; Wang et al., 2023b), we employ CLIP-based (Radford et al., 2021) scores and DINO-based (Caron et al., 2021) scores to measure the text alignment and style conformity. Following EvalCrafter (Liu et al., 2023), we measure the temporal consistency of video generation by (i) calculating clip scores between contiguous frames and (ii) calculating the warping error on every two frames with estimated optical flow. Note that these metrics are not perfect. For example, one can easily achieve a close-to-1 style score by entirely replicating the style reference. Similarly, stylized results may yield inferior text scores compared to realistic results, even though both accurately represent the content descriptions. We recommend a comprehensive consideration of both CLIP-based text scores and style scores, rather than relying solely on a single metric.

In addition to quantitative analysis, we conducted a user study to make comparisons among our method, VideoCrafter, Gen-2, and AnimateDiff in the context of single-reference and multi-reference stylized video generation. Users are instructed to select their preferred option based on style conformity, temporal quality, and all options fulfill text alignment for each comparison pair. We randomly chose 15 single-reference pairs and 10 multi-reference pairs, collecting 1125 votes from 15 users. Further details can be found in the supplementary materials.

VideoComposer (Wang et al., 2024) is a controllable video generation model that allows multiple conditional inputs including style reference image. It is a natural competitor of our method. Besides, we construct two additional comparative methods, i.e. VideoCrafter* and Gen2*, which extend VideoCrafter (Chen et al., 2023a) and Gen2 (Gen-2, 2023), the state-of-the-art T2V models in open-source and close-source channels respectively, to make use of style reference images by utilizing GPT-4V (OpenAI, 2023) to generate style prompts from them. The quantitative comparison is tabulated in Table 2. Several typical visual examples are illustrated in Figure 5.

We can observe that: (i) VideoComposer tends to copy content from style references and struggles to generate text-aligned content, which is possibly because of the invalid decoupling learning. Consequently, its results exhibit abnormally high style conformity and very low text alignment. In addition, VideoComposer often generates static videos, thus having the lowest warping errors, but this does not mean that their results perform best in temporal quality. (ii) VideoCrafter* exhibits limited stylized generation capabilities, producing videos with diminished style and disjointed movements. Gen-2* demonstrates superior stylized generation capabilities. However, Gen-2 is still limited by the inadequate representation of style in textual descriptions, and is more prone to sudden changes in color and luminance. (iii) In comparison, our method captures styles more effectively and reproduces them in the generated results.

AnimateDiff (Guo et al., 2024) denotes a paradigm to turn personalized SD (i.e., SD fine-tuned on specific-domain images via LoRA (Hu et al., 2022) or Dreambooth (Ruiz et al., 2023)) for video generation, namely combined with pre-trained temporal blocks of T2V models. It can generate very impressive results if the personalized SD is carefully prepared, however, we find it struggle to achieve as satisfactory results if only a handful of style reference images are available for training. We conduct an evaluation on 60 text-style pairs with multi-references, as presented in Sec. 4.1. We train Dreambooth (Ruiz et al., 2023) models for each style and incorporate them into AnimateDiff based on their released codebase. Thanks to the flexibility of Q-Former, our method also supports multiple reference images in a tuning-free fashion, i.e. computing the image embeddings of each reference image and concatenating all embeddings as input to the Q-Former. Results are compared in Table 3 and Figure 6 respectively.

According to the results, AnimateDiff struggles to achieve high-fidelity stylistic appearance while tends to generate close-to-realism results despite the style references are typical artistic styles. In addition, it is vulnerable to temporal artifacts. As the trained personalized-SD can generate decent stylistic images (provided in the supplementary materials), we conjecture that the performance degradation is caused by the incompatibility from the pre-trained temporal blocks and independently trained personalized-SD models, which not only interrupts temporal consistency but also weakens the stylistic effect. In contrast, our method can generate temporal consistent videos with high style conformity to the reference images and accurate content alignment with the text prompts. Furthermore, using multiple references can further promote the performance, which offers additional advantages in practical applications.

We first study the effectiveness of content-style decoupled data augmentation. As depicted in Table 4, training with the original image-caption pairs restricts the model’s ability to extract style representations, leading to lower style conformity. For example, as shown in Figure 7, method without data augmentation fails to capture the ”3D render” style from the reference.

As discussed in Sec. 3.1, we make a comparison between attach-to-text and dual cross-attention to study their effects. Results are presented in Table  4 and Figure 7, revealing that attach-to-text tends to directly fuse the content from the reference image and the text prompts rather than combining the text-based content and image-based style. This indicates the effectiveness of dual cross-attention in facilitating content-style decoupling.

As previously discussed in Figure 3, our proposed adaptive style-content fusion module demonstrates effectiveness in adaptively processing various conditional contexts. It benefits the generalization ability of model to deal with diverse combinations of content prompt and style image. Figure 8 reveals that although the baseline can handle easy prompt inputs like ”A little girl”, it struggles to accurately generate all objects described in longer prompts. In contrast, the adaptive fusion module can achieve decent text alignment for long text descriptions thanks to its flexibility to adaptive balance between content and style.

Our proposed training scheme consists of two stages, i.e., style adapter training and temporal adaption. To show its necessity, we build two baselines: (i) w/o Temporal Adaption: that we train a style adapter on image data and apply it directly to stylized video generation without finetuning; (ii) joint training: that we conduct style adapter training and temporal blocks finetuning on image-video dataset simultaneously. As depicted in Table 5, baseline (i) exhibits inferior temporal consistency when applied directly to video, and undermines the content alignment and style conformity. As for baseline (ii), the learning of style embedding extraction seems to be interfered by the joint finetuning of temporal blocks, which impedes it to generate desirable stylized videos.

Dreambooth (Ruiz et al., 2023) aims to generate images of a specific concept (e.g., style) by finetuning the entire text-to-image model on one or serveral images. We train Dreambooth based on Stable Diffusion 1.5. The training prompts are obtained from BLIP-2 (Li et al., 2023a), and we manually add a style postfix using the rare token ”sks”. For example, ”two slices of watermelon on a red surface in sks style” is used for the first style reference in Table S3. We train the model for 500 steps for single-reference styles and 1500 steps for multi-reference styles, with learning rates of $5\times 10^{-6}$ and a batch size of $1$. The training steps are carefully selected to achieve the balance between text alignment and style conformity.

InST (Zhang et al., 2023) propose a inversion-based method to achieve style-guided text-to-image generation through learning a textual description from style reference. We train InST for 1000 steps with learning rates of $1\times 10^{-4}$ and a batch size of $1$.

We extend Stable Diffusion to style-guided text-to-video gerneration by utilizing GPT-4v to generate style descriptions from style reference. Details about style descriptions can be found in Table S3

IP-Adapter (Ye et al., 2023) propose to train an image-conditioned adapter to generate images from image prompts. We use the official checkpoint of IP-Adapter-Plus(SDXL) for evaluation. Note that IP-Adapter is primarily designed for image variants and other editing tasks. When conducted with its default scale value $s=1$, IP-Adapter tends to simply reconstruct style references, which actually underestimates the ability of IP Adapters in stylized generation. During the evaluation, we adjust the scale value to 0.5 to ensure a more balanced comparison.

Style Aligned (Hertz et al., 2023) design a self-attention sharing mechanism to ensure constant style among different samples, supporting both stylized generation and style transfer tasks. We conduct the official implementation on SDXL during the evaluation.

Similar to SD*, We use VideoCrafter (Chen et al., 2023a) $320\times 512$ Text2Video Model and Gen-2 (Gen-2, 2023) equipped with GPT-4v to generate stylized videos from style references and text prompts.

AnimateDiff (Guo et al., 2024) aims to extend personalized T2I model(i.e., Dreambooth or LoRA (Hu et al., 2022)) for video generation. To compare with AnimateDiff, we first train personalized dreambooth models for each group of multi-reference style images, then we incorporate them into AnimateDiff based on their released codebase. We did not use LoRA because we observed that AnimateDiff fails to turn LoRA-SD for video generation in most cases.

We utilize GPT4 to generate prompts across four meta-categories: human, animal, object, and landscape. Initially, 15/10 prompts (for images/videos) were generated in each category. Recognized as low-quality prompts, semantically repeated prompts, containing style descriptions, and other less informative ones were manually filtered, leading to 5/3 prompts per category. For video prompts, we specifically encouraged the generation of scenarios involving motion. The final prompts in testset are provided in Table S1 and Table S2.

We collect 20 diverse single-reference stylized images and 8 sets of style images with multi-reference(each contains 5 to 7 images in similar styles) from the Internet¹¹1The style references are collected from https://unsplash.com/, https://unsplash.com/, https://en.m.wikipedia.org/wiki/, https://civitai.com/, https://clipdrop.co/. Besides, for the comparison with the Text-to-Image model including Stable Diffusion and the Text-to-Video model including VideoCrafter and Gen-2, we extend them to stylized generation by equipped them with GPT-4v to generate textual style descriptions from style reference. We provide style references and corresponding style descriptions in Table S3 and Figure S1.

We utilize the pretrained CLIP-ViT-H-14-laion2B-s32B-b79K²²2https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K as a feature extractor(also for CLIP-Style and CLIP-Temp), which is trained on LAION-2B and demonstrates enhanced performance across various datasets. We extract the frame-wise image embeddings from the generated results and text embeddings from the input content prompts, then compute their average cosine similarity. The overall CLIP-Text is calculated as:

where $M$ represents the total number of testing videos and $T$ represents the total number of frames in each video($T=1$ for image generation), $emb(x_{t}^{i})$ and $emb(p^{i})$ indicate the CLIP embedding of the $t$-th frame of the $i$-th video $x_{t}^{i}$ and the corresponding prompt $p^{i}$, respectively.

Similarly, we extract the frame-wise image embeddings $emb(x_{t}^{i})$ from the generated results and image embeddings $s^{i}$ from the input style reference. The overall CLIP-Text is calculated as:

Considering the semantic consistency between every two frames, we extract the frame-wise CLIP image embeddings and compute the cosine similarity between each of the two frames, as follows:

For the warping error, we first obtain the optical flow between each two frames using RAFT-small(Teed and Deng, 2020), a pre-trained optical flow estimation network. Subsequently, we compute the pixel-wise differences between the warped image and the predicted image, as follows:

where $warp(x_{t+1}^{i},flow(x_{t+1}^{i},x_{t}^{i}))$ represents the warped frame of $x_{t+1}^{i}$ using the optical flow between frame $x_{t+1}^{i}$ and frame $x_{t}^{i}$.