FusionFrames: Efficient Architectural Aspects for
Text-to-Video Generation Pipeline

Prior works on video generation utilize VAEs [20, 2, 3, 47, 39], GANs [38, 23, 18, 16, 5], normalizing flows [15] and autoregressive transformers [42, 43, 8, 13, 36]. GODIVA [42] adopts a 2D VQVAE and sparse attention for T2V generation. CogVideo [13] is built on top of a frozen CogView2 [6] T2I transformer by adding additional temporal attention layers.

Recent research extend T2I diffusion-based architecture for T2V generation [29, 11, 4, 9, 54]. This approach can benefit from pretrained image diffusion models and transfer that knowledge to video generation tasks. Specifically, it introduces temporal convolution and temporal attention layers interleaved with existing spatial layers. This adaptation aims to capture temporal dependencies between video frames while achieving computational efficiency by avoiding infeasible 3D convolutions and 3D attention mechanisms. These temporal layers can be trained independently [29, 4] or jointly [11, 7] with the 2D spatial layers. This technique of mixed spatial-temporal blocks for both convolutions and attention layers has become widespread in most T2V models [12, 29, 11, 7, 44, 4, 54, 17]. Alternative approaches to operating with the time dimension include the use of an image diffusion model in conjunction with a temporal autoregressive Recurrent Neural Network (RNN) to predict individual video frames [48], projection of 3D data into a latent 2D space [49] and the use of diffusion to generate latent flow sequences [21]. In this paper, we propose separate temporal blocks as a new approach for conditioning temporal information.

Finally, it is worth noting that certain studies [29, 11] operate entirely in pixel space, whereas others [9, 54, 44, 7, 4, 52, 17] utilize the more efficient latent space. We follow the second approach in this paper.

MCVD [37] uses a diffusion-based model for interpolation. It leverages four consecutive keyframes (two from each side) to predict three frames in between. In the field of T2V research, diffusion-based architectures are commonly adopted for interpolation. Some studies make use of training methods involving Masked Frame Interpolation [11, 29, 4, 50]. In LVDM [9], they use Masked Latent Clip Interpolation instead. VideoGen [17] employs a flow-based approach for interpolation. Magicvideo [54] combines the latents of two consecutive keyframes with randomly sampled noise. It is worth noting that several training techniques have been proposed, including Conditional Frame (or Latent) Perturbation [11, 9, 50] to mitigate the error introduced by the previous generation step and Context Guidance (or Unconditional Guidance) [4, 9] to enhance the diversity and fidelity of sampled video.

In our interpolation network, we adjust the input and output layers in U-Net to generate three identical frames before training. Then, we train U-Net to predict three interpolated frames (between each two consecutive keyframes). This adaptation significantly reduces the computational cost compared to Masked Frame Interpolation methods, enabling pre-trained T2I weights for initialization.

In their works, [4, 17] builds a video decoder with temporal attention and convolution layers. MagicVideo [54], on the other hand, incorporates two temporal-directed attention layers in the decoder to build a VideoVAE. To the best of our knowledge, previous studies have not compared their strategies for constructing a video decoder. In this research, we present multiple options for designing a video decoder and conduct an extensive comparative analysis, evaluating their performance regarding quality metrics and the implications for additional parameters.

The keyframes generation is based on a pretrained latent diffusion T2I model Kandinsky 3.0 [1]. We use the weights of this model to initialize the spatial layers of the keyframe generation model, which is distinguished by the presence of temporal components. In all experiments, we freeze the weights of T2I U-Net and train only temporal components. We consider two fundamental ways of introducing temporal components into the general architecture – temporal layers of convolutions with attention integrated into spatial blocks and our separate temporal blocks. Figure 2 exhaustively explains our concept.

We also investigate different types of temporal conditioning in convolution and attention layers. In the case of convolution, we use its 3D version with kernels $3\times 1\times 1$ and $3\times 3\times 3$ corresponding to Temporal Conv1D and Temporal Conv3D, respectively. A similar mechanism was implemented for temporal attention layers to build interaction not only between latent positions between each other across time dimension (Temporal Attn1D) but also between a window of positions across the video (Temporal Attn3D) (Figure 2 - right). We used an attention window of size $2\times 2\times T$ in Temporal Attn3D, where $T$ is the total number of generated frames. The algorithm of dividing the whole frame into groups of four pixels is the same as Block Attention in MaxViT [34].

In our case, separate temporal blocks are organized in three ways:

• •

  Conv1dAttn1dBlocks – Incorporation of a Temporal Conv1D Block following each spatial convolution block and a Temporal Attn1D Block following each self-attention block in the T2I model.

• •

  Conv3dAttn1dBlocks – Incorporation of a Temporal Conv3D Block following each spatial convolution block and a Temporal Attn1D Block following each self-attention block. Additionally, two extra Linear layers are introduced within the Temporal Conv3D Block to downsample the hidden dimension of the input tensor and subsequently upsample it to the original size. These projections are employed to maintain a comparable number of parameters as in the Conv1dAttn1dBlocks configuration.

• •

  Conv1dAttn3dBlocks – Incorporation of a Temporal Conv1D Block following each spatial convolution block and a Temporal Attn3D Block following each self-attention block. In this configuration, no extra parameters are introduced compared to Conv1dAttn1dBlocks, as the alteration is limited to the sequence length in attention calculation. Moreover, there is minimal computational overhead due to the relatively low number of keyframes (16 for all our models)..

In the case of the mixed spatial-temporal block (Conv1dAttn1dLayers), we exclusively employ the most standard form of integration, incorporating Temporal Conv1D and Temporal Attn1D layers into the inner section of the spatial blocks (Figure 2 - left). We made this choise because the integration style represented by Conv1dAttn1dLayers consistently yields worse results across all metrics when compared to Conv1dAttn1dBlocks.

We apply interpolation in the latent space to predict a group of three frames between each pair of consecutive keyframes. This necessitates the adaptation of the T2I architecture (Figure 3). First, we inflate the input convolution layer (by zero-padding weights) to process: (i) A group of three noisy latents representing interpolated frames (${z_{t}=[z_{t}^{1},z_{t}^{2},z_{t}^{3}];z_{t}^{i}\in R^{4\times 32\times 32}}$), and (ii) two conditioning latents representing keyframes (${c=[c^{1},c^{2}];c^{i}\in R^{4\times 32\times 32}}$). The input to U-Net consists of the channel-wise concatenation of $z_{t}$ and $c$ (${[z_{t},c]\in R^{20\times 32\times 32}}$). Similarly, we inflate the output convolution layer (by replicate padding weights) to predict a group of three denoised latents (${z_{t-1}=[z_{t-1}^{1},z_{t-1}^{2},z_{t-1}^{3}];z_{t-1}\in R^{12\times 32\times 32}}$). Secondly, we insert temporal convolution layers $\phi$ in the original T2I model $\theta$. This enables upsampling a video with arbitrary length $T\rightarrow 4T-3$ frames. The activation derived from a temporal layer $out_{temporal}$ is combined with the output of the corresponding spatial layer $out_{spatial}$ using a trainable parameter $\alpha$ as follows:

A detailed description of these adjustments is provided in Appendix 9. Finally, we remove text conditioning, and instead, we incorporate skip-frame $s$ (section 4) and perturbation level $tp$ conditioning (Check below for details). For interpolation, we use v-prediction parameterization ($v_{t}\equiv\alpha_{t}\epsilon-\sigma_{t}x$) as described in [28, 11].

Building on the conditional frame perturbation technique [11, 4, 9, 50], we randomly sample a perturbation level (number of forward diffusion steps) $pt\in\{0,1,2,\ldots,250\}$ and use it to perturb the conditioning latents $c$ using forward diffusion process (Section 3). This perturbation level is also employed as a condition for the U-Net (as described earlier).

With some probability $uncond\_prob$, we also replace the conditioning latents $c$ with zeros for unconditional generation training. The final training objective looks like:

where $m=0$ specify that we replace conditioning frames with zeros and $m=1$ otherwise. We employ context guidance [4] at inference, with $w$ representing the guidance weight:

Our interpolation model can benefit from a relatively low guidance weight value, such as $0.25$ or $0.5$. Increasing this value can significantly negatively impact the quality of the interpolated frames, sometimes resulting in the model generating frames that closely match the conditional keyframes (No actual interpolation).

To enhance the video decoding process, we use a pretrained MoVQ-GAN [53] model with a frozen encoder. To extend the decoder into the temporal dimension, we explore several choices: Substituting 2D convolutions with 3D convolutions and adding temporal layers interleaved with existing spatial layers. Regarding the temporal layers, we explore the use of temporal convolutions, temporal 1D conv blocks, and temporal self-attentions. All additional parameters are initialized with zeros.

In this section, we provide a comparison of our trained models using FVD, IS on UCF-101 and CLIPSIM on MSR-VTT, as detailed in Table 1. First, we assess the models trained for $100k$ training steps. Concerning the comparison with Conv1dAttn1dLayers, our results clearly indicate that the inclusion of temporal blocks, rather than temporal layers, results in significantly enhanced quality according to these metrics. In the assessment of temporal blocks, Conv3dAttn1dBlocks falls behind, achieving values of $594.919$ for FVD and $22.899$ for IS. Conv3dAttn1dBlocks shows an IS score of $23.381$ for IS and an FVD score of $573.569$. Comparing with Conv3dAttn1dBlocks, Conv3dAttn1dBlocks records a worse IS score of $23.063$ but a the best FVD score $545.184$. The CLIPSIM results do not show a clear difference among temporal blocks, with slightly better performance observed for Conv1dAttn3dBlocks, reaching a CLIPSIM of $0.2956$.

Next, we showcase the best outcomes achieved for Conv1dAttn1dBlocks after being trained for $220k$ training steps. Evidently, our model experiencing significant improvements in terms of FVD ($433.504$) and IS ($24.325$).

We want to highlight that the relatively lower performance in IS compared to the baselines could be attributed to ambiguities in IS assessment. Existing literature lacks sufficient details regarding the methodologies used for calculating IS on video in related works.

We performed a preference test to compare our models between each other. For this purpose, we created a bot that displays a pair of videos. An annotator then chooses the video with best quality in terms of three visual measurements: 1) frame quality, 2) alignment with text, and 3) temporal consistency. A total of $31$ annotators engaged in this process and the total number of used video pairs is $6600$. Figure 5 presents the results of this user study. There is a clear preference of videos generated by temporal blocks based models over the videos generated by temporal layer based model with a large margin among all measurements. The temporal layer based model often produces either semantically unrelated keyframes, or does not deal well with the dynamics. Regarding the comparison between temporal block methods against each other, there is a small but consist preference for the model with Temp. Conv1D Block and Temp. Attn3D Block over the two other models. Finally, there is no clear preference between the rest two models. The qualitative comparison reveals visually observable advantages of our technique, both in the quality of generated objects on individual keyframes and in dynamics as illustrated in Figure 4. The method based on temporal blocks generates more consistent content and more coherent keyframes sequence. More examples with video generation can be found on the project page and in supplementary material.

To assess the effectiveness of our interpolation architecture, we implemented Masked Frame Interpolation (MFI) architecture described in Video LDM [4] and Appendix 9. For our interpolation architecture, we use a $220k$ training steps checkpoint for a model trained with temporal 1D-Blocks. We generated interpolated frames for a set of $2048$ videos, all composed exclusively of generated keyframes. Results, as presented in Table 2, reveal that our new interpolation architecture excels in generating interpolated frames with superior quality and higher fidelity. Specifically, it achieves FVD of $433.054$ compared to $550.932$ obtained with MFI. The inference time associated with each architecture is illustrated in Figure 6, demonstrating that our new architecture is more efficient, reducing the running time to less than third compared to MFI.

We conducted comprehensive experiments, considering many choices of how to build video decoder, and assessed them in terms of quality metrics and the impact on the number of additional parameters. The results are presented in Table 3. This evaluation process guided us in making the optimal choice for production purposes. Extending the decoder with a $3\times 3\times 3$ temporal convolution and incorporating temporal attention during the fine-tuning process yields the highest overall quality among the available options. In this manner, we apply finetuning to the entire decoder, including spatial layers and the newly introduced parameters. An alternative efficient choice involves using a $3\times 1\times 1$ temporal convolution layer or temporal 1D Block with temporal attention, which significantly reduces the number of parameters from $556M$ to $220M$ while still achieving results that closely match the quality obtained through the more extensive approach.