Aesthetic Attributes Assessment of Images with AMANv2 and DPC-CaptionsV2

Image Aesthetic Quality Assessment (IAQA) is the aesthetic assessment of images. In the last decades, IAQA is very popular in the fields of computer vision, computational aesthetics, psychology and neuroscience. Aesthetic quality assessment also provides important reference for image quality assessment (IQA) in some scenarios (such as photography and post-editing).

There are two categories in most researches: high aesthetic quality (professional) and low aesthetic quality (amateur). The second popular assessment task is to give a continuously numerical score of aesthetics. Besides, numerical assessment task is to predict a score distribution of human rating in aesthetics is [10, 17, 30].

For a human artist, when shown a photo or a drawing, he/she will not just give a numerical score but always say a paragraph to describe many aesthetic attributes such as composition, lighting, color, focus of the image. In the image, not only the objects and the edge need to be detected, higher-dimensional image features, such as the proportion of the object, the collocations with fashion and the camera techniques, are more important manifestations, and captions can show them well. Zhou et al. [5] construct the AVA-Comments dataset with only visual features. They improve the performance of aesthetics assessment on the AVA dataset but giving comments for an image. Pioneer work of Chang et al. [6] proposes aesthetic captioning of images. They build Photo Critique Captioning Dataset (PCCD) for the community. The PCCD contains 4,235 images and 29,645 comments. Each image is attached with comments and scores of 7 aesthetic attributes. However, they only output a sentence of assessment, which can not give a full review of aesthetic attributes. The value of PCCD is not fully explored. Besides, the size of PCCD is relatively small compared to AVA dataset [29], which is commonly used in this field but do not contain ground truth of aesthetic captions and attributes. Jin et al. [1] proposed the DPC-Caption data set, and for the first time combined image captions of multiple attributes with the task of aesthetic quality evaluation.

In this work, we propose Aesthetic Attributes Assessment of Images, as shown in Figure 1. We predict aesthetic attributes captions and the aesthetic score of each attribute. Based on the previous work, we build a better dataset, named DPC-CaptionsV2 by the aesthetic knowledge transferring method from dpchallenge.com. DPC-CaptionsV2 contains comments up to 5 aesthetic attributes of one image. There are 112,000 images and 560,000 comments. Then, we propose aesthetic multi-attribute network version 2, which contains a base image captioning network (such as bottom-up and top-down attention network and object-semantics transformer layers) and aesthetic features fusion network. We train this model on both small-scale PCCD dataset (4,235 images and 29,645 comments), containing attribute comments and scores, and our large-scale DPC-CaptionsV2 dataset with only attribute comments. We evaluate our method of captioning of attributes on DPC-CaptionsV2 with image captioning criteria.

As we know, this is the first work that can produce captions for aesthetic attributes of an image. In the early version of this paper [1], we proposed method Aesthetic Multi-Attribute Network (AMAN) and dataset DPC-Captions. Constructing the dataset depends on the proposed method: transferring from a small-scale image dataset with full annotations to the large-scale dataset with weak annotations. We extend the previous work badly, and the main differences include:

• •

  In this paper we propose DPC-CaptionsV2 (92,006 images and 392,625 comments). The aesthetic attributes are classified into four main attributes of photography: composition, lighting, color, subject. We update the comment matching algorithm from simple keyword matching in DPC-Captions to Bag of Words (BoW) with a BERT-based text classification model. That reduces the bias of BoW to aesthetic comments in PCCD. Using the new matching algorithm, for each attribute, there are up to 5 comments that can be matched. We choose one of them as the ground truth caption and the others as reference ones, which make the output captions more stable. Compared with DPC-Captions, our new version provides more extensive aesthetic captions, which greatly improves the quality of the dataset.

• •

  We propose Aesthetic Multi-Attribute Network version 2 (AMANv2), which uses a two-stage training processes on a small-scale full annotated dataset and a large-scale weakly annotated one. Smaller datasets are used to filter the required aesthetic comments from the original data. On the larger DPC-CaptionV2 dataset, we improve Bottom-Up and Top-Down Attention (BUTD) and Visual Language Pre-training and Self-Attention (VLPSA), and adds small-scale, based embedding, layers to the model. The aesthetic attributes of the data set can predict the model regression on the required target. Based on aesthetic multi-attributes and the above attention networks, we propose Aesthetic Multi-Attribute Network version 2 (AMANv2).

Image Aesthetic Quality Assessment. Before deep learning era, many hand-crafted features [9, 18] are designed for aesthetic image classification and scoring as surveyed by Deng et al. [11]. Deep learning methods are proposed recently for aesthetic assessment[13, 16, 17, 19, 20, 22, 23, 25, 26, 34]. They outperform traditional methods. Lu et al. [23] propose a two column CNNs for binary classification. Mai et al. [26] introduce ratio-preserving assessment of aesthetics by using SPP. Kong et al. [22] propose the AADB dataset which contains scores of 12 aesthetic attributes and use a rank-preserving loss for aesthetic scoring. Kao et al. [19] propose an aesthetics assessment combing the semantic classification of a image. Jin et al. [17] propose CJS-CNN for aesthetic score distribution prediction. In the above literatures, they only consider numerical assessment without taking the aesthetic assessment by languages into consideration. Other methods are also used for image aesthetic evaluation, such as Pfister et al. [36] uses self-supervised learning to replace common ImageNet-based pre-training models. Regarding the composition’s info of the image, Liu et al. [37] propose a full convolutional network as the feature encoder of the input image, and uses the encoded feature map to represent the composition information in the image, and further graph convolution infers the image. Zeng et al. [38] proposed a unified probability formula for three image aesthetics evaluation tasks (binary classification, average score regression and score distribution prediction), and improved the noisy original score distribution to obtain a data distribution with stronger generalization ability. Kong et al. [31] proposed a method of UAV video quality assessment.

New tasks and datasets of IAQA. Kong et al. [22] design a dataset named AADB (Aesthetics and Attributes Database) which contains 8 aesthetic attributes of each image. However, the label of each aesthetic attribute is only binary value (good or bad). Zhou et al. [35] design a dataset named AVA-Comments, which adds comments from DPChallenge.com to AVA dataset [29] which only contains aesthetic score distributions of images. Zhou et al. use the image and the attached comments to give a binary classification of aesthetics. Wang et al. [33] design a dataset named AVA-Reviews, which selects 40,000 images from AVA dataset and contains 240,000 reviews. Chang et al. [6] design PCCD dataset, which contains 4,235 images and 29,645 comments. However, both [33] and [6] can only give a single sentence as the comments of the aesthetics of an image. They do not describe the individual aesthetic attributes. [45] proposed a probabilistic aesthetic caption-filtering method for cleaning internet data to generate a larger dataset: AVACaptions,which has more higher quality captions than PCCD. [46] proposed the task of food image aesthetic captioning and decomposed it into two tasks: single-aspect captioning and an unsupervised text compression. They also collect a dataset which contains comments related to six aesthetic attributes.

Image Captioning. Most work of image captioning follow CNN-RNN framework and achieve great results [12, 21, 27]. Most of recent literatures of image captioning [7, 4, 3, 24, 28] introduce attention scheme. We follow this trend and add attention model in our network.

Recent studies [39, 40, 41, 42] on vision-language pretraining (VLP) have shown that it can effectively learn generic representations from massive image-text pairs, and that fine-tuning VLP models on task-specific data achieves state-of-the-art (SoTA) results on well-established visual and language tasks. The latest research results are OSCAR (Object-Semantics Aligned Pre-training)[43] and CLIP (Contrastive Language-Image Pre-training)[44]. CLIP is instead focused on learning visual models from scratch via natural language supervision and does not densely connect the two domains with a joint attention model. Any language tag input can be used as a tag for image classification and regression analysis.

During the data cleaning process of the DPC-Caption version 1, we get help from PCCD dataset [6]. The aesthetic attributes of PCCD dataset include Color Lighting, Composition, Depth of Field, Focus, General Impression and Use of Camera. For each aesthetic attribute, the five most frequent keywords are selected from the captions. We omit the adverbs, prepositions and conjunctions. We merge words of similar meaning such as color and colour, color and colors. A statistic of the keywords frequency is shown in Table I.

Counting the frequency of few words causes that the definition of image aesthetics is limited by a few words and lacking diversity. Due to using too many words in the selecting process, models will be required to adjust the weight of a few high-frequency words in order to filter out real and effective comments.

In our attempts, the text classification and clustering model will not be able to separate the adjectives describing the aesthetics of the image from the specific objects. For example, both blue and sky appear once in blue sky, but only the color description word blue is related to the aesthetic of the image, instead of the the word sky.

We use Bow (Bag of Words) and a BERT-based text classification model to filter out the necessary comments. Among them, Bow uses the 1,000 most frequently occurring words or phrases in the PCCD data set, and removes some nouns and stopwords. We count the vocabulary word frequency in each sentence as the aesthetic weight and rank, and take the top 100,000 comments. In the BERT-based text binary classification model, we use all PCCD comments as positive samples, and randomly select the same number of 29,645 comments from the COCO data set as negative samples. We judge whether a sentence meets the aesthetic attributes. After satisfying the Bow weight ranking, it still has an aesthetic classification result.

Compared with version 1, DPC-CaptionsV2 has the more keywords for the classification of aesthetic captions, such as composition, color, light, and subject. as tags. The tags of captions we used contain more vocabulary related to aesthetic attributes. After two screening weeks, there are still 92,006 DPC subtitle images. In the absence of image-related photography attributes or use of camera, it is difficult for the subtitles to learn relevant features directly from the image. So we deleted the image related to the use of camera attribute. We also found that the keywords, composition, depth of field and focus are similar, so we merged them. Finally, we obtained the 4 attributes of DPC-CaptionsV2, as shown in the table I. The number of images with light is 22,034. And, the number of images with subject is 14,306, and so on.

In subsequent experiments, the experiment shows that the task of image captioning in the aesthetics is more difficult to learn and generate than the task of general image captioning. Based on this, we adjust the captions’ length in the DPC-CaptionsV2 dataset to make it shorter than DPC-Captions. At the same time, we let each caption has at least 3 reference comments, so we can get easier in training process and validation. Comparing our DPC-CaptionsV2 with the PCCD, AVA-Reviews, AVA-Comments and DPC-Caption data sets in the table II, it can be seen that AVA-Reviews and AVA-Comments do not contain aesthetics Attributes. Compared with DPC-Captios, DPC-CaptionsV2 provides more authentic and extensive aesthetic captions to improve the quality of the dataset.

The PCCD dataset includes captions and attribute scores, while the DPC-Captions dataset only includes captions. In the previous research, for the image in DPC-Captions, only some of the 5 attributes may have annotations attached. We think that PCCD is a fully annotated dataset, and DPC-Captions is a weakly annotated dataset. We recommend learning from a combination of fully annotated datasets and weakly annotated datasets.

Rethinking the process of dataset construction, we find that using a small dataset to guide the construction of larger datasets is improper. It is easy to produce both over-fitting on the results and the limitations of captions. Therefore, we used a larger dataset to guide the generation of DPC-Caption: fine-tuned to generate aesthetic features with a pre-trained model based on ImageNet and Visual Genome.

In this paper, we propose a new task of IAQA: aesthetic attributes assessment. A new vision-language dataset called DPC-Captions is built by knowledge transfer for this task. We propose a novel network AMANv2 for two-stage learning processes on both full annotated small-scale dataset and weakly annotated large-scale dataset, which based on Bottom-Up and Top-Down Attention (BUTD) and Visual Language Pre-training and Self-Attention (VLPSA). Our AMANv2 can generate captions of individual aesthetic attributes.

In the future, we will explore to caption from sentences to paragraphs. The knowledge transfer methods can be used to build larger dataset for weakly supervised learning. The relations among attributes can be used for caption learning. Reinforcement learning can also be leveraged for captions generation.

Parts of this work has been appeared in our previous conference version [1]. This work is partially supported by the National Natural Science Foundation of China (62072014), the Beijing Natural Science Foundation (L192040), and the CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ-2021-022A).