Human-centric Image Cropping with Partition-aware and Content-preserving Features

The flowchart of the proposed method is illustrated in Figure 3, in which we adopt a similar pipeline as [44, 23]. Given an image, we first integrate multi-scale feature maps from a pretrained backbone (\eg, VGG16 [34]) to obtain the basic feature map. After that, we update the basic feature map to the partition-aware feature map, based on which we extract partition-aware region feature and content-preserving feature for each candidate crop. Finally, we predict the crop-level score using the concatenation of partition-aware region feature and content-preserving feature.

To acquire the human bounding box, we leverage Faster R-CNN [31] trained on Visual Genome [19] to detect human subjects for human-centric images. We check each image to ensure that the predicted bounding box correctly encloses the main human subject. We describe how to determine the main subject and discuss the robustness of our method against human detection in Supplementary. As illustrated in Figure 1, the whole image can be divided into nine non-overlapping partitions by the human bounding box. We conjecture that the aesthetic value of similar content in different partitions often varies with its relative position to the human subject, so the feature map should be partition-aware.

To achieve this goal, we derive partition-aware feature map from the basic feature map, as illustrated in the left subfigure in Figure 3. Given an $H\times W$ basic feature map $\mathbf{M}$ with $C$ channels, we partition it into nine regions $\{\mathbf{M}_{k}|_{k=1}^{9}\}$ with $\mathbf{M}_{k}$ being the $k$-th partition. Considering that the relative position of each partition to human subject is conditioned on the human information (\eg, face orientation and posture), as exemplified in Figure 1, we extract human feature $\mathbf{f}^{h}$ using RoIAlign [14] to implicitly encode the aforementioned human information and reduce its dimension to $\frac{C}{2}$, which is appended to each location of the basic feature map. The resultant feature map is represented by $\hat{\mathbf{M}}$ with $\hat{\mathbf{M}}_{k}$ being the $k$-th partition. To explicitly tackle different partitions differently, we employ nine individual nonlinear transformations for each partition to update their features with residual learning by

$$\mathbf{F}_{k}=\Phi_{k}(\hat{\mathbf{M}}_{k})+\mathbf{M}_{k},$$

(1)

where we use a $3\times 3$ convolutional (conv) layer with $C$ output channels followed by $\mathrm{ReLU}$ as the transformation function $\Phi_{k}(\cdot)$.

After that, we obtain a new feature map $\mathbf{F}$ by combining all updated partitions $\{\mathbf{F}_{k}|_{k=1}^{9}\}$, which has the same size as the basic feature map $\mathbf{M}$. By integrating human feature and employing partition-specific transformations, similar contents in different partitions can produce different responses conditioned on the human information. Thus, we refer to $\mathbf{F}$ as partition-aware feature map. Following [43, 23], given a candidate crop, we employ RoIAlign [14] and RoDAlign [43] schemes to extract its partition-aware region feature based on $\mathbf{F}$, denoted as $\mathbf{f}^{r}$.

Apart from the position of the human in the crop and human information, the preservation of important content also plays an important role when evaluating candidate crops. So we propose to predict a heatmap to indicate the location of important content and then automatically learn content-preserving features to augment our method.

To model the relation between pairwise regions, we define the adjacency matrix $\mathbf{A}\in\mathcal{R}^{L\times L}$ according to the cosine similarity of region features following [33]. Then we perform reasoning over the graph $\mathbf{\bar{F}}$ by graph convolution [32]:

$$\mathbf{\bar{F}}^{\prime}=\sigma(\mathbf{A}\mathbf{\bar{F}}\mathbf{\Theta}),$$

(2)

where $\mathbf{\Theta}\in\mathcal{R}^{C\times C}$ is the the trainable weight matrix of the graph convolution layer and $\sigma(\cdot)$ is $\mathrm{ReLU}$ activation. Then, we reshape $\mathbf{\bar{F}}^{\prime}$ back to $\mathbf{F}^{\prime}\in\mathcal{R}^{H\times W\times C}$. Compared with the conventional convolution, graph convolution allows the message flow across local regions, which is helpful for important content prediction (see Section 4.6).

There is no ground-truth heatmap for important content. Nevertheless, existing cropping datasets [43, 41] are associated with multiple scored crops for each image, with a larger score indicating higher aesthetic quality. Based on the assumption that highly scored crops are more likely to contain important content, we propose to generate pseudo ground-truth heatmap from the weighted average of highly scored crops.

Specifically, given an image with multiple candidate crops, we suppose the score of the $m$-th crop to be $y_{m}$. We take the average score of all crops in the dataset as the threshold to select highly scored crops, and convert the bounding box of each selected crop to a binary map, which is resized to the same size as the predicted heatmap $\widetilde{\mathbf{H}}$. We obtain the pseudo ground-truth heatmap $\mathbf{H}$ via the weighted average of all binary maps, in which larger weight is assigned to the crop with higher score. Here, we perform $\mathrm{softmax}$ normalization to produce the weights for each highly scored crop: $\omega_{m}=\frac{\mathrm{exp}(y_{m})}{\sum^{N_{h}}_{m=1}\mathrm{exp}(y_{m})}$, in which $N_{h}$ is the number of highly scored crops. In the training stage, we employ an L1 loss to supervise the heatmap learning:

$$\mathcal{L}_{cont}=\|\mathbf{H}-\widetilde{\mathbf{H}}\|_{1}.$$

(3)

As demonstrated in Figure 2, the pseudo ground-truth heatmaps can highlight the attractive objects in the background and the objects that person interacts with, which are often ignored by saliency detection methods.

Finally, for each candidate crop, we concatenate its partition-aware region feature $\mathbf{f}^{r}$ and content-preserving feature $\mathbf{f}^{c}$, which is passed through a fully connected (fc) layer to get the aesthetic score for this crop.

We train the proposed model with a multi-task loss function in an end-to-end manner. Given an image containing $N$ candidate crops, the ground-truth and predicted scores of the $m$-th crop are denoted by $y_{m}$ and $\tilde{y}_{m}$, respectively. We first employ a smooth L1 loss [31] for the score regression considering its robustness to outliers:

$$\mathcal{L}_{reg}=\frac{1}{N}\sum^{N}_{m=1}\mathcal{L}_{s1}(y_{m}-\tilde{y}_{m}),$$

(4)

where $\mathcal{L}_{s1}(\cdot)$ represents the smooth L1 loss:

$$\mathcal{L}_{s1}(x)=\left\{\begin{array}[]{cc}0.5x^{2},&\mathrm{if}\ |x|<1,\\ |x|-0.5,&\mathrm{otherwise}.\end{array}\right.$$

(5)

Besides regression loss, we also use a ranking loss [23] to learn the relative ranking order between pairwise crops explicitly, which is beneficial for enhancing the ability of ranking crops with similar content. With $e_{m,n}=y_{m}-y_{n}$ and $\tilde{e}_{m,n}=\tilde{y}_{m}-\tilde{y}_{n}$, the ranking loss is computed by

$$\mathcal{L}_{rank}=\frac{\sum_{m,n}\max\big{(}0,\mathrm{sign}(e_{m,n})(e_{m,n}-\tilde{e}_{m,n})\big{)}}{N(N-1)/2}.$$

(6)

After including the heatmap prediction loss in Eqn.(3), the total loss is summarized as

$$\mathcal{L}=\mathcal{L}_{reg}+\mathcal{L}_{rank}+\lambda\mathcal{L}_{cont},$$

(7)

in which the trade-off parameter $\lambda$ is set as 1 via cross-validation (see Supplementary).

Limited by the small number of annotated human-centric images in existing image cropping datasets [43, 41], only training on human-centric images would lead to weak generalization ability to the test set. Therefore, we employ both human-centric and non-human-centric images to train our model. For non-human-centric images, we use the basic feature map $\mathbf{M}$ to replace the partition-aware feature map $\mathbf{F}$, because there could be no dominant subject used to partition the image. Besides, we extract content-preserving features from non-human-centric images in the same way as human-centric images, because preserving important content is also crucial for non-human-centric images. In this way, our model is able to train and infer on both human-centric images and non-human-centric images.

We conduct experiments on the recent GAICD dataset [43], which contains 1,236 images (1,036 for training and 200 for testing). Each image has an average of 86 annotated crops. There are 339 and 50 human-centric images in the training set and test set of GAICD dataset, respectively. As described in Section 3.4, we employ the whole training set (1,036 images) for training and evaluate on the human-centric samples of the test set. Following [43], three evaluation metrics are employed in our experiments, including the average Spearman’s rank-order correlation coefficient ($\overline{SRCC}$) and averaged top-$N$ accuracy ($\overline{Acc_{N}}$) for both $N=5$ and $N=10$. The $\overline{SRCC}$ computes the rank correlation between the ground-truth and predicted scores of crops for each image, which is used to evaluate the ability of correctly ranking multiple annotated crops. The $\overline{Acc_{N}}$ measures the ability to return the best crops.

Apart from GAICD dataset, we also collect 176 and 39 human-centric images from existing FCDB [5] and FLMS [11] datasets, respectively. We evaluate our method on the collected 215 human-centric images from two datasets, following the experimental setting in [28, 37]: training the model on CPC dataset [41], and using intersection of union (IoU) and boundary displacement (Disp) for performance evaluation. Note that we train on the whole CPC dataset, which contains 10,797 images including 1,154 human-centric images and each image has 24 annotated crops.

Following existing methods [15, 37, 23], we use VGG16 [34] pretrained on ImageNet [8] as the backbone. We apply $1\times 1$ conv to unify the channel dimensions of the last three output feature maps as $256$ and add up three feature maps to produce the multi-scale feature map. We reduce the channel dimension of multi-scale feature map to 32 using a $1\times 1$ conv, \ie, $C=32$. $E^{g}$ is implemented by two $3\times 3$ convs and pooling operations followed by a fc layer. The dimensions of partition-aware region feature $\mathbf{f}^{r}$ and content-preserving feature $\mathbf{f}^{c}$ are both $256$. Similar to [23, 43], the short side of input images is resized to 256 and the aspect ratios remain unchanged. We implement our method using PyTorch [30] and set the random seed to 0. More implementation details can be found in Supplementary.

In this section, we start from the general pipeline of existing methods [44, 23] and evaluate the effectiveness of two types of features. The results are summarized in Table 1. In the baseline (row 1), we only use the region feature extracted from the basic feature map $\mathbf{M}$ to predict scores for each crop.

For fair comparison, we use the pretrained VGG16 [34] as the backbone for all baselines and train them on GAICD dataset, based on their released code or our own implementation. For our method, we additionally report the results of a basic version (“Ours(basic)”) without using partition-aware feature or content-preserving feature (row 1 in Table 1). It can be seen that Ours(basic) yields similar results with GAIC(ext) because they adopt the same region feature extractor (RoI+RoD). Among the baselines, GAIC(ext)* [44] and CGS [23] are two competitive ones, owning to the more advanced architecture and the exploitation of mutual relations between different crops. Finally, our model outperforms all the state-of-the-art methods, which demonstrates that our method is more well-tailored for the human-centric image cropping task.

Apart from GAICD dataset [43], we also collect 176 and 39 human-centric images from existing FCDB [5] and FLMS [11] datasets, respectively, and compare our method with the state-of-the-art methods on these two datasets in Table 3. Following [41, 37], we train the model on CPC dataset [41], and use IoU and Disp as evaluation metrics. Additionally, we adopt the strategy in [28] to generate candidate crops and return the top-1 result as best crop without extra post-processing for all methods except CACNet [15], which is trained to regress the best crop directly. As shown in Table 3, our proposed model produces better results, but the performance gain is less significant than that on GAICD dataset. As claimed in [43], one possible reason is that the IoU based metrics used in FCDB and FLMS datasets are not very reliable for evaluating cropping performance. Furthermore, we also evaluate our method on both human-centric and non-human-centric images, and present results in Supplementary.

To take a close look at the impact of partition-aware feature on candidate crop evaluation, we use the region features extracted from basic feature map and partition-aware feature map to predict scores for crops, respectively, corresponding to row 1 and row 2 in Table 1. As shown in Figure 5, to ensure that crop pairs have different aesthetic value yet similar content, for each image, we generate crop pair by moving its ground-truth best crop horizontally or vertically, in which the new crop still contains the human subject. We can see that using partition-aware feature consistently leads to larger and more reasonable score changes than basic feature despite the various face orientations or postures of the human in Figure 5, which is beneficial for correctly ranking crops with similar content.

The ablation study in Section 4.3 demonstrates the superiority of graph-based relation mining (“GCN”) over the conventional convolution (“Conv”) when predicting the heatmap of important content (see row 7,8 in Table 1). To reveal their difference qualitatively, we show the source image, its pseudo ground-truth heatmap, the heatmap predicted by “conv”/“graph” convolution in Figure 6. With GCN learning the mutual relation between different regions, the model can make a more reasonable estimation of important content, especially the border area. For example, in the source image in the first row, we show an unpleasant outer area in the red dashed box, which should be removed for composing a good crop. The unimportant content (low values in the heatmap) predicted by “GCN” completely covers the unpleasant area, while “Conv” only covers part of the unpleasant area. In the second row, unlike “Conv” that only deems the area around person as important, “GCN” predicts relatively high values for the area behind person, indicating that preserving such area in a crop may be beneficial. In summary, “GCN” can facilitate important content localization and contributes to more informative content-preserving feature.