Paragraph-to-Image Generation with Information-Enriched Diffusion Model

Prior works [5, 3] directly utilize frozen encoder-decoder language model (i.e., T5 [25]) for short-text to image generation tasks, restricting the text encoder to within 128 tokens. We present a insight that efficiently fine-tuning a more powerful decoder-only language model can yield stronger performance in long-text alignment (up to 512 tokens). Compared to the encoder-decoder architecture of large language models, recent decoder-only models, such as ChatGPT [22], and Llama V2 [35], have garnered more attention and success. There are some evident advantages to using the decoder-only language models as text encoders: 1) Models like Llama and the GPT series, based on the decoder-only architecture, exhibit stronger semantic understanding and generalization capabilities. 2) The training corpora are also nearly four times larger than that of the encoder-decoder language model, i.e., T5 [25]. However, typically, decoder-only architectures are not adept at feature extraction and mapping tasks. Therefore, we propose Paragraph-Image Alignment Learning with Language Model Adaptation to address this issue, adapting the decoder-only model, i.e., Llama V2 [35] for text-to-image generation tasks.

Given a text description $y$, the standard Text to Image Diffusion Models is probabilistic model designed to model conditional distributions of the form $p(z|y)$, where a conditional denoising autoencoder $\epsilon_{\theta}(z_{t},t,y);\,t\in{\{1,\dots,T\}}$ is used to learn the reverse process of a fixed Markov Chain of length $T$. The corresponding objective can be simplified to:

where $\tau_{\theta}$ denotes the text encoder. $\mathcal{E}$ refers to a AE [27] for mapping image to latent feature. Typically, during training, we optimize only the Unet of the diffusion model $\epsilon_{\theta}$, while freezing the $\tau_{\theta}$. This is because we consider the text embedding from $\tau_{\theta}$, i.e., CLIP, to be well-suited for text-image alignment tasks. However, when using an LLM as the text encoder, it is essential to consider whether a frozen LLM is appropriate for this context.

Inspired by the success of instruction-tuning [36, 16] and LoRA [11], we propose a strategy for paragraph-image alignment learning with language model adaptation. This involves freezing the pretrained Large Language Model weights $\tau_{\theta_{0}}$ and introducing a certain degree of trainable parameters $\Delta\theta(\Theta)$ to learn the textual relationships between paragraphs and images. The objective is revised as follows:

where both $\epsilon_{\theta}$ and $\tau_{\Delta\theta(\Theta)}$ can be jointly optimized for learning better text representation. Compared to direct fine-tuning, this strategy offers two advantages: 1) During the training of paragraph-image alignment, it preserves the powerful semantic understanding capabilities of the LLM, preventing knowledge overfitting to simple text-image semantics. 2) Storage and compute efficient. Requires only limited computational resources and incurs no additional inference costs.

As shown in Figure 2, ParaDiffusion adopts a three-stage training approach. Similar to the prior works [4, 30], stage 1 is employed to acquire general text-image semantic alignment knowledge. Stage 2 is introduced to simultaneously fine-tune the LLM and the diffusion model for paragraph-image alignment. In Stage 3, a high-quality small dataset, consisting of $3k$ carefully selected images, is used to further enhance the model performance.

Stage 1: Pre-train for Short-text and Image Alignment In this stage, we gathered a dataset of 300 million examples to train the Unet of diffusion model with 1.3 billion parameters. Among these, one billion examples were selected from the 5 billion images of the LAION dataset [32]. Additionally, we constructed an internal dataset containing over 200 million examples. Similar to SDXL [23], the model was trained with progressively increasing resolutions. It started with training at a resolution of 256 and was later adjusted to a resolution of 512. The entire pre-training process took 5 days on 56 A100 GPUs.

Stage 2: Paragraph-Image Alignment Learning with LLM Adaptation. To perform paragraph-image alignment learning (§ 3.1.2), we construct a large scale dataset with several million paragraph-image pairs, namely ParaImage-Big, where the long-form text is generated by CogVLM [37], as illustrated in Figure 3(a). The entire paragraph-image alignment learning process took 2 days on 56 A100 GPUs.

Stage 3: High-Quality Alignment Data Finally, we created an extremely small but high-quality dataset, namely ParaImage-Small, to further enhance the performance of model with a small learning rate. This dataset consists of 3 manually selected images, each accompanied by human-annotated long-text descriptions. These descriptions provide detailed information about the objects in the images, including their attributes and spatial relationships.

Inspired by the success of current large vision-language models [37, 43, 19], we propose to leverage the SOTA model, i.e., CogVLM for automatically generating extensive long-form textual annotations for images. Firstly, we gathered approximately 3.3 million high-quality images from the LAION-Aesthetics [15] and SAM datasets [14]. For the LAION-Aesthetics, we downloaded around $8$ million images with aesthetics scores above $6$ and then filtered them to a set of $3.2$ million images with a minimum short edge resolution of 512 pixels. For SAM dataset, we obtained 2 million images and further filtered out those with mosaic elements, resulting in a final dataset of around 100k images. Then, we prompted CogVLM to generate corresponding long-form textual descriptions for the images, as illustrated in Figure 3(a). Figure 4 and Table 1 present a detailed statistical comparison, and it is evident that in terms of textual descriptions, the captions of the proposed dataset are richer and longer in semantic content. $92\%$ of captions from LAION have a text length of fewer than 25 words, whereas over $70\%$ of captions from ParaImage-Big exceed $100$ words, with a few extending to over 200 words.

The generated captions from CogVLM cannot be guaranteed to be $100\%$ accurate; therefore, it is essential to create a small but high-quality dataset with manual annotations (i.e., ParaImage-Small). This dataset consists of $3$k, manually selected by annotators from $650k$ exquisite images of LAION-Aesthetics, adhering to common principles in photography. Then, the $3k$ images were annotated with corresponding long-text descriptions, detailing the objects in the images, the attributes of these objects, and their spatial relationships. The details for Paragraph Description Annotation Rule are as follows:

1. 1.

   The description should be no less than $50$ words and no more than $500$ words.

2. 2.

   The first sentence of the description should commence with a portrayal of the main subject of the image, e.g., A close-up photo of a beautiful woman.

3. 3.

   The description should be objective and avoid subjective emotions and speculations, such as (‘On a cloudy day, it might rain in the future’).

4. 4.

   The description should cover details about the objects’ number, color, appearance, and location.

5. 5.

   The description should follow a logical sequence, such as from left to right or from the center outward.

6. 6.

   Avoid using prefixes like ‘In the picture is,’ ‘This image depicts’, or ‘Here is.’

Details regarding the selection principles of images are provided in the supplementary materials. Based on the images selection annotation rules, we invited and trained $10$ annotators to score the images from $1$ to $5$ based on aesthetics and content, with $5$ being the highest score. There are two rounds of quality inspection, each round involving $3$ staff members for review. Finally, we selected images with the highest score, resulting in around $3000$ images. With these images, we hired $20$ annotators to provide a detailed description for each image following the annotation guidelines above. Then two rounds of quality audits were conducted afterwards, with two people in each round. For the images that does not meet the evaluation criteria, we would conduct revisions. The entire labeling process took two weeks, with one week for selecting pictures and the other week to provide detailed text descriptions.

Zero-shot FID-30K on MS-COCO $256\times 256$ [18] is a universal evaluation method for text-image generation tasks. Table 2 presents relevant comparisons between ParaDiffusion and other existing works. Our ParaDiffusion achieved an FID score of $9.64$, demonstrating similar performance to SD XL [23] and PIXART-$\alpha$ [3]. In comparison, RAPHAEL [40] and DeepFloyd-IF [5] achieved better scores, while utilizing larger models with more parameters. Furthermore, we would like to argue that FID may not be an appropriate metric for image quality evaluation, where a higher score does not necessarily indicate better-generated images. Many studies [3, 5, 23], have demonstrated that, instead, the evaluation by human users is a more authoritative measure.

Following the prior works [3, 6], we also conducted a User Study evaluation from the perspectives of visual appeal and text faithfulness on 300 prompts from ViLG-300 [6]. We only chose the recent models with sota performance for comparative evaluation, as involving human evaluators can be time-consuming. Figure 7 presents the related rating proportions from expert evaluators.

Visual Appeal. Our model significantly outperforms DeepFloyd-IF and SD XL-Refiner in terms of visual appeal, while both of these are considered among the best models of the past year. In comparison to PIXART-$\alpha$[3], our model has achieved competitive performance. We want to emphasize that our model did not specifically focus on Visual Appeal. Additionally, PIXART-$\alpha$[3] and our work are concurrent efforts. Therefore, we believe the performance is acceptable.

Text Faithfulness. Figure 7 (right) shows that our model achieved outstanding performance in Text Faithfulness among the four models, with a voting percentage of $13.7\%$. ‘Tie (Same Performance)’ received a very high voting percentage. This is because ViLG-300 includes many simple prompts, leading to consistently good results and making it challenging to differentiate among them. We provide additional cases in the supplementary material to further analyze this situation.

Figure 8 presents the results on ParaPrompts-400.

Visual Appeal. Similar to ViLG-300 [6], our ParaDiffusion achieved outstanding results in terms of visual appeal on the ParaPrompts dataset, surpassing SD XL and DeepFloyd-IF and approaching PIXART-$\alpha$ [3]. Compared to the performance on the ViLG-300 [6] dataset, there is a decrease in the voting percentage for the ‘Tie’ category. This indicates that in more challenging long-text scenarios, the performance differences among different models become more pronounced.

Text Faithfulness. As for paragraph-guided image generation setting, ParaDiffusion demonstrates a more pronounced advantage in text faithfulness, reaching $60.3\%$, while other models achieve only around $10\%$. This indicates that ParaDiffusion significantly outperforms other models in aligning long-text aspects such as object, object attributes, and object positional relationships. In the supplementary material, we provide additional visualizations and analyses to support this claim.

Table 3 presents the ablation study for LLM Adaptation. ‘Base’ denotes directly performing paragraph-image alignment learning (Stage-2) with ParaImage-Big without fine-tuning Llama V2 using LoRA. Compared to the base model, our LLM adaptation with LoRA demonstrates an improvement of nearly $5\%$ in human voting rates. Another insight reveals that during the process of increasing trainable parameters from $4.2$ million to $67.1$ million, the performance appears to gain consistently. Therefore, in our final configuration, we randomly select 16.7 million trainable parameters as the ultimate setting. Figure 9 (b) presents the visual comparisons for the performance from LLM adaptation.

Table 4 presents the ablation study for the proposed dataset, ParaImage. ParaImage-Big brings significant improvements, with approximately a $50\%$ increase in human voting rates for both visual appeal and text faithfulness simultaneously. The smaller-scale, and high-quality set of 3k images in ParaImage-Small further enhances aesthetic aspects, achieving a remarkable around $70\%$ increase in human preference rates compared to that of Stage 2. Figure 9 (a) presents a visual comparison, vividly illustrating the gains in visual appeal. In this case, the prompt is ‘An exquisite sculpture in the center of a square on a rainy day.’