DiVE: DiT-based Video Generation with Enhanced Control

Spatial View-Inflated Attention. To guarantee the multi-view consistency during generation, we replace the commonly used cross-view attention modules [4, 8] with a parameter-free view-inflated attention mechanism. Specifically, we extend 2D spatial self-attention to enable cross-view interactions by reshaping the input from $B~{}\times~{}V~{}\times~{}T~{}\times H^{\prime}~{}\times W^{\prime}~{}\times~{}C$ to $B~{}\times~{}T~{}\times~{}(VH^{\prime}W^{\prime})~{}\times~{}C$ and treating $VH^{\prime}W^{\prime}$ as the sequence length. Consequently, our proposed approach improves the multi-view coherence without compensating with additional parameters.

Caption-Layout Joint Cross-Atttention. Following MagicDrive [4], we use a cross-attention mechanism to integrate scene captions and layout entries. Whereas the layout entries, i.e. instance details such as 2D coordinates, heading and ID, are Fourier-encoded and combined into a unified embedding. Instance captions are encoded using a pre-trained CLIP [12] model. These embeddings are concatenated and processed through an MLP, producing the final layout embedding, which, along with the scene caption embedding, conditions the cross-attention mechanism.

ControlNet-Transformer. Delving into details, we introduce ControlNet-Transformer to ensure the precision towards the road sketch control inspired by PixArt-$\delta$ [3]. Practically, a pre-trained VAE extracts the latent features from road sketches, which are then processed by a 3D patch embedder for the sake of consistency issue with our main network. To parameterize our mentioned design, 13 duplicated blocks are integrated with the first 13 base blocks with the DiT [11] architecture. Each duplicated block combines the road sketch features and base block outputs, using spatial self-attention to reduce the computational overhead.

Variable Resolution and Frame Length. Following OpenSora [16], we adopt the Bucket strategy, which ensures that videos within each batch have a consistent resolution and frame length.

Rectified Flow. Inspired by OpenSora 1.2 [16], we replace IDDPM [9] with rectified flow [7] during the later training stages for increased stability and reduced inference steps. Rectified flow, an ODE-based generative model, defines the forward process between data and normal distributions as

$$x_{t}=(1-t)x_{0}+tx_{1}~{},$$

(1)

where $x_{1}$ is a data sample, and $x_{0}$ is a sample from the normal distribution. The loss function is constructed as

$$\ell(\theta):=\mathbb{E}_{x_{1},x_{0}}[\left\|\upsilon_{\theta}(x_{t},t,c)-(x_{1}-x_{0})\right\|^{2}_{2}]~{},$$

(2)

with $c$ encompassing the three conditions. Sampling is performed from $t=1$ to $t=0$ in $N$ steps via

$$x_{t-\frac{1}{N}}=x_{t}-\frac{1}{N}\upsilon_{\theta}(x_{t},t,c),\forall t\in\left\{1,2,...,N\right\}/N~{}.$$

(3)

First-$\bm{k}$ Frame Masking. To enable arbitrary-length video generation, we propose a first-$k$ frame masking strategy, allowing the model to seamlessly predict future frames from the preceding ones. Formally, given a binary mask $m$ indicating the frames to be masked—where the unmasked frames serve as the condition for future frame generation—we update $x_{t}$ as

$$x_{t}\leftarrow x_{t}\odot(1-m)+x_{1}\odot m~{},$$

(4)

with losses calculated only on unmasked frames. During inference, video is generated autoregressively, with the last-$k$ frames of the previous clip conditioning the next.

Classifier-free Guidance for Multi-Conditions. We observe that extending classifier-free guidance from the text condition to layout entries and road sketches enhances conditional control precision and visual quality. During training, we set the text condition $c_{T}$, layout condition $c_{L}$, and sketch condition $c_{R}$ to $\phi$ with a $5\%$ probability each, and also enforce a $5\%$ probability where all three conditions are simultaneously set to $\phi$. The guidance scales $\lambda_{T}$, $\lambda_{L}$, $\lambda_{R}$ correspond to the scene caption, layout entries, and road sketch, respectively, and measure the alignment between the sampling results and conditions. Inspired by [1] , the modified velocity estimate is as follows:

	$\displaystyle\upsilon^{\prime}_{\theta}$	$\displaystyle=\upsilon_{\theta}(x_{t},\phi,\phi,\phi)$		(5)
		$\displaystyle+\lambda_{T}\cdot(\upsilon_{\theta}(x_{t},c_{T},c_{L},c_{R})-\upsilon_{\theta}(x_{t},\phi,c_{L},c_{R}))$		(6)
		$\displaystyle+\lambda_{L}\cdot(\upsilon_{\theta}(x_{t},\phi,c_{L},c_{R})-\upsilon_{\theta}(x_{t},\phi,\phi,c_{R}))$		(7)
		$\displaystyle+\lambda_{R}\cdot(\upsilon_{\theta}(x_{t},\phi,\phi,c_{R})-\upsilon_{\theta}(x_{t},\phi,\phi,\phi))~{}.$		(8)

Dataset and Evaluation Metrics. We train and evaluate our model using the nuScenes [2] dataset and the interpolated $12$Hz annotations provided by challenge. The generated multi-view videos are assessed based on distribution similarity (FVD), temporal consistency (DTC [10] and CTC [12]), visual quality (MUSIQ [5]) and controllability. Controllability is evaluated through two perception tasks: 3D object detection and BEV segmentation, with BEVFormer [6] serving as the perception model.

Training Details. We train our method in four stages using eight NVIDIA A800 GPUs. In the first stage, we fine-tune on OpenSora 1.1 [16] checkpoints with fixed-resolution images of $512\times 512$ for $30k$ steps to control layout and sketch, training the ControlNet-Transformer, spatial attention, and layout net with spatial self-attention in base blocks. In the second stage, we train the model for $26k$ steps with variable resolutions (144p, 240p, 360p) and frame lengths to adapt to the nuScenes dataset, continuing to use spatial self-attention. The final two stages replace IDDPM with rectified flow, training for $20k$ steps at 144p to 360p, then $80k$ steps at higher resolutions (480p to full).

Inference Details. We perform sampling inference using rectified flow with $30$ steps, choosing 480p resolution for a balance between inference time and visual quality. Each inference round uses a frame length of $16$. We set $\lambda_{L}$ and $\lambda_{R}$ to $2.0$, adjusting $\lambda_{T}$ to $1.0$ for night scenes and $7.0$ for other scenes to achieve the best results.

To assess the quality of generated videos, we compare our method with the challenge baseline, MagicDrive [4], using evaluations on 16-frame sequences. As shown in Table 1, our model outperforms MagicDrive in terms of data distribution similarity, temporal consistency, visual quality, and controllability. Additionally, Figure 2 illustrates that the videos produced by our model exhibit both higher visual quality and better spatial consistency. Figure 3 demonstrates the scene editing capability of our method, where the weather in the generated video changes according to the caption, while other objects remain unchanged.

Effect of Proposed Classifier-free Guidance. We compared different classifier-free guidance methods, both with and without unconditional layout and sketch considerations, as detailed in Table 2. The ”Score” is calculated as in the 1st round of the challenge, with CFG_T,L,R being our proposed method. Excluding unconditional sketch (CFG_T,L) or both (CFG_T) yielded slightly better FVD but showed more pronounced differences in BEV segmentation and 3D object detection. We also evaluated CFG_MagicDrive from MagicDrive [4], which performed well in controllability but had only satisfactory FVD. Ultimately, CFG_T,L,R achieved the best overall score.