AIGCIQA2023: A Large-scale Image Quality Assessment Database for AI Generated Images: from the Perspectives of Quality, Authenticity
and Correspondence

We adopt six latest text-to-image generative models, including Glide[24], Lafite[34], DALLE[26], Stable-diffusion[27], Unidiffuser[1], Controlnet[33], to produce AIGIs by using open source code and default weights. To ensure content diversity and catch up with the practical application requirements, we collect diverse texts from the PartiPrompts website [32] as the prompts for AIGI generation. The text prompts can be simple, allowing generative models to produce imaginative results. They can also be complex, which raises the challenge for generative models. We select 10 scene categories from the prompt set, and each scene contains 10 challenge categories. Overall, we collect 100 text prompts (10 scene categories $\times$ 10 challenge categories) from PartiPrompts[32]. The distribution of the selected scene and challenge categories is displayed in pie chart of Fig.1. It can be observed that the dataset exhibits a high level of scene diversity, with images generated covering a broad range of challenges. Then we perform the text-to-image generation based on these models and prompts. Specifically, for each prompt, we generate 4 various images randomly for each generative model. Therefore, the constructed AIGCIQA2023 database totally contains 2400 AIGIs (4 images $\times$ 6 models $\times$ 100 prompts) corresponding to 100 prompts.

Subjective IQA is the most reliable way to evaluate the visual quality of digital images perceived by the users. It is generally used to construct image quality datasets and served as the ground truth to optimize or evaluate the performance of objective quality assessment metrics. Due to the unnatural property of AIGIs and different text prompts having different target image spaces, it is unreasonable to just use one score, i.e., “quality” to represent human visual preferences. In this paper, we propose to measure the human visual preferences of AIGIs from three perspectives including quality, authenticity, and text-image correspondence. For an image, these three visual perception perspectives are related but different.

The first dimension of AIGI evaluation is “quality” evaluation, i.e., evaluating an AIGI from its clarity, color, lightness, contrast, etc., which is similar to the assessment of NSIs. During the experiment procedure, subjects are instructed to evaluate whether the image outline is clear, whether the content can be distinguished, and the richness of details, etc. Fig.3 (a) shows 10 high quality examples and 10 low quality examples of the images generated by the prompt of “a corgi”.

Considering the generation nature of AIGIs, an important problem of these images is that they may not look real compared to NSIs. Therefore, we introduce a second dimension of evaluation metrics for the generated images, i.e., “authenticity” evaluation. For this dimension, subjects are instructed to assess the image from the authenticity aspect, i.e., whether it looks real or whether they can distinguish that the image is AI-generated or not. Fig.3 (b) shows 10 high authenticity and 10 low authenticity examples of images generated by the prompt of “a girl”.

Since an AIGI is generated from a text, it is also important to evaluate its correspondence with the original prompt, i.e., the third dimension, text-image “correspondence”. For this purpose, subjects are instructed to consider textual information provided with the image and then give the correspondence score from 0 to 5 to assess the relevance between the generated image and its prompt. Fig.3 (c) shows 10 high text-image correspondence and 10 low correspondence examples of images generated by the prompt of “a grandmother reading a book to her grandson and granddaughter”.

To evaluate the quality of the images in the AIGCIQA2023 and obtain Mean Opinion Scores (MOSs), a subjective experiment is conducted following the guidelines of ITU-R BT.500-14 [3]. The subjects are asked to rate their visual preference degree of exhibited AIGIs from the quality, authenticity and text-image correspondence. The AIGIs are presented in a random order on an iMac monitor with a resolution of up to 4096 × 2304, using an interface designed with Python Tkinter, as shown in Fig.4. The interface allows viewers to browse the previous and next AIGIs and rate them using a quality scale that ranges from 0 to 5, with a minimum interval of 0.01. A total of 28 graduate students (14 males and 14 females) participate in the experiment, and they are seated at a distance of around 60 cm in a laboratory environment with normal indoor lighting.

We follow the suggestions recommended by ITU to conduct the outlier detection and subject rejection. The score rejection rate is 2%. In order to obtain the MOS for an AIGI, we first convert the raw ratings into Z-scores, then linearly scale them to the range $[0,100]$ as follows:

where $r_{ij}$ is the raw ratings given by the $i$-th subject to the $j$-th image. $N_{i}$ is the number of images judged by subject $i$.

Next, the mean opinion score (MOS) of the image j is computed by averaging the rescaled z-scores as follows:

where $MOS_{j}$ indicates the MOS for the $j$-th AIGI, $M$ is the number of valid subjects, and $z^{\prime}_{i}{}_{j}$ are the rescaled z-scores.

To further illustrate the differences of the three perspectives, we demonstrate several example images and their corresponding subjective ratings from three aspects in Fig.5. For each subfigure, it can be noticed that the right AIGI outperforms the left AIGI on two evaluation dimensions but is much worse than the left AIGI on another dimension, which demonstrates that each evaluation perspective (quality, authenticity, or text-image correspondence) has its own unique perspective and value.

Fig.6 demonstrates the MOS and score distribution for quality evaluation, authenticity evaluation, and text-image correspondence evaluation, respectively, which demonstrate the images in AIGCIQA 2023 cover a wide range of perceptual quality.

Since the AIGIs in the proposed AIGCIQA2023 database are generated based on text prompts and have no pristine reference images, they can only be evaluated by no-reference (NR) IQA metrics. In this paper, we select fifteen state-of-the-art IQA models for comparison. The selected models can be classified into two groups:

• •

  Handcrafted-based models, including: NIQE[23], BMPRI[21], BPRI[19], BRISQUE[22], HOSA[30], BPRI-LSSn[19], BPRI-LSSs[19], BPRI-PSS[19], QAC[31], HIGRADE-1 and HIGRADE-2[16].

  These models extract handcrafted features based on prior knowledge about image quality.

• •

  Deep learning-based models, including: CNNIQA[13], WaDIQaM-NR[2], VGG (VGG-16 and VGG-19)[28] and ResNet (ResNet-18 and ResNet-34)[11].

  These models characterize quality-aware information by training deep neural networks from labeled data.

In this study, we utilize the following four performance evaluation criteria to evaluate the consistency between the predicted scores and the corresponding ground-truth MOSs, including Spearman Rank Correlation Coefficient (SRCC), Pearson Linear Correlation Coefficient (PLCC), Kendall’s Rank Correlation Coefficient (KRCC), and Root Mean Squared Error (RMSE).

All the benchmark models are validated on the proposed AIGCIQA2023 database. For traditional handcrafted-based models, they are directly evaluated based on the database. For deep trainable models, we first randomly split the database into an 4:1 ratio for training/testing while ensuring the image with the same prompt label falls into the same set. The partitioning and evaluation process is repeated several times for a fair comparison while considering the computational complexity, and the average result is reported as the final performance. For deep learning-based models, we applied CNNIQA[13], WaDIQaM-NR[2], VGG (VGG-16 and VGG-19)[28] and ResNet (ResNet-18 and ResNet-34)[11] to predict the MOS of image quality. The repeating time is 10, the training epochs are 50 with an initial learning rate of 0.0001 and batch size of 4.

The performance results of the state-of-the-art IQA models mentioned above on the proposed AIGCIQA2023 database are exhibited in Table 1, from which we can make several conclusions:

• •

  The handcrafted-based methods achieve poor performance on the whole database, which indicates the extracted handcrafted features are not effective for modeling the quality representation of AIGIs. This is because most employed handcrafted features of these methods are based on the prior knowledge learned from NSIs, which are not effective for evaluating AIGIs.

• •

  The deep learning-based methods achieve relatively more competitive performance results on three evaluation perspectives. However, they are still far away from satisfactory.

• •

  Most of the IQA models achieve better performance on quality evaluation and worse on text-image correspondence score assessment. The reason is that the text prompts for image generation are not utilized for the IQA model training. This makes it more challenging for the IQA models to extract relation features from AIGIs, which inevitably leads to performance drops.