3D View Optimization for Improving Image Aesthetics

Contemporary studies on automatic composition editing for aesthetic enhancement predominantly focus on cropping techniques that trim input images. Such conventional cropping methods are constrained to adjusting the inner regions of an image for editing, particularly when the primary subject is close to the image’s boundary, thus limiting the scope for aesthetic improvements. Zhong et al. [25] overcame this limitation by integrating a pre-cropping extrapolation of the area outside the original frame, thereby expanding the search space for aesthetic optimization.

Another method of changing the composition of an image is through image retargeting [3, 19], which resizes an image without distortion while preserving the shape of the main subject. Liu et al. [15] proposed a method that automatically performs retargeting while moving the salient subjects in the image to more aesthetically pleasing positions.

While existing approaches to enhancing image aesthetics predominantly employ 2D image processing techniques, our method ventures into an unexplored domain to improve the aesthetics of input images through 3D scene exploration; we optimize camera position, gazing direction, field-of-view angle, and image aspect ratios, all the while preserving the 3D structure of the captured scene.

Traditional photo-editing tools struggle to incorporate 3D scene information, often failing to preserve spatial relationships between subjects during composition editing. CompZoom [4] addresses this by allowing manual adjustments to subject size and position through changes in focal length and camera position, utilizing a multi-view camera model and requiring multiple images from different focal lengths and positions of the same scene. Conversely, Liu et al. [16] introduced a composition-editing tool that leverages depth information from a single image, enabling adjustments in size, position, and perspective of objects while maintaining the scene’s 3D structure, albeit necessitating a degree of expertise in photographic composition and familiarity with the tool for effective use. Our approach simplifies this process, offering an automatic composition editing method that manipulates camera parameters within a 3D scene reconstructed from a single image, making it accessible to users without in-depth photography knowledge.

Prior research on aesthetic view search within 3D spaces has predominantly utilized reinforcement learning to ascertain optimal camera positions and orientations in both real and virtual 3D environments. Xie et al. [22] introduced a technique for generating sequences of aesthetically appealing camera postures within a pre-constructed 3D scene, employing the same aesthetics evaluation model as AlZayer et al. [2]. In contrast, our approach focuses on identifying aesthetic views in the 3D scene reconstructed from a single input photograph, offering a novel perspective on enhancing image aesthetics.

The task of image aesthetics evaluation involves categorizing images into distinct levels of aesthetics or quantifying an image’s beauty with an aesthetics score. The advent of deep learning and the availability of extensive datasets for aesthetics assessment have led to deep learning-based methods outperforming traditional, manually crafted approaches in terms of accuracy and efficiency. These methods, noted for their robustness and minimal inference costs, have been effectively integrated into view-search algorithms within 3D spaces, as discussed in Section 2.3. Our approach leverages the aesthetics evaluation model called a View Evaluation Net (VEN) proposed by Wei et al. [21], which calculates aesthetics scores for images generated from initial viewpoints and various camera positions. This model, trained on a comprehensive dataset comprising over 1 million photo-to-picture pair comparisons, excels in determining the relative aesthetics among pairs of image crops, providing a solid foundation for our method’s aesthetics-based view selection.

If we naïvely reconstruct a 3D scene from the input image, a slight change in camera position or gazing direction will reveal out-of-photographed regions beyond the edges of the input image. To avoid this, we extrapolate the input image in advance. Specifically, the input image is first uniformly resized such that the larger dimension, whether height $h$ or width $w$ of the input image, measures 512 pixels. Subsequently, we expand the image’s borders by 256 pixels on all sides (i.e., top, bottom, left, and right) utilizing Adobe Photoshop’s generative fill function (without text prompts). This preparatory step not only enhances the search space for aesthetic improvements but also effectively minimizes the visibility of areas beyond the original photographic range.

Unfortunately, the abovementioned image extrapolation is not perfect yet; a large change in camera translation or gazing direction may still reveal out-of-photographed regions.

To further suppress the appearance of such regions, our method integrates a regularization term defined with a binary mask rendered under new camera parameters. Let $\mathcal{P}$ be the colored point cloud obtained from the input image, $\mathbf{\Theta}$ be the camera parameters, and $\mathcal{M}$ be the rendering function that outputs a binary mask. The regularization term is defined as follows:

where $\mathbf{1}_{w\times h}$ is an all-“1” image of $w\times h$ pixels.

Let $\mathcal{R}$ be a rendering function that outputs a color image from point cloud $\mathcal{P}$ under the camera parameter $\Theta$, and $\mathcal{A}$ be an evaluation function that outputs an aesthetic score from a color image. We update the camera parameter $\Theta$ using the following equation.

where $\lambda_{\mathit{mask}}$ is the weight of $\mathcal{L}_{\mathit{mask}}$ and we set $\lambda_{\mathit{mask}}=10$.

Initially, we used the differentiable renderer of PyTorch3D [11] as the rendering function $\mathcal{R}$ and the Adam optimizer [12] to optimize Equation (2). However, our preliminary experiment showed that this optimization easily falls into local minima. Therefore, we adopt a black-box optimization algorithm called CMA-ES [8] for a more global search. Note that this algorithm does not require gradient computation and thus differentiable rendering is not required.

The camera parameters to be optimized in this study are the camera translation vector $\mathbf{t}=(t_{x},t_{y},t_{z})$, camera rotation angle $\mathbf{\theta}=(\theta_{\mathit{roll}},\theta_{\mathit{pitch}},\theta_{\mathit{yaw}})$, and vertical field-of-view angle $\theta_{\mathit{fovy}}$. Furthermore, to change the aspect ratio of the output image, we introduce coefficients $s_{w},s_{h}$ to control the output image size $(s_{w}w)\times(s_{h}h)$ and optimize these coefficients. A comparison with and without optimization of these coefficients is presented in the next section.

We implemented our method using Python, PyTorch, PyTorch3D, and Optuna [1]. Experiments were conducted using the test images of the following three datasets: FCDB [5], FLMS [7], and GAICD [23]. These three datasets contain 309, 500, and 500 test images, respectively. Image extrapolation was performed on all of the test images in advance. The optimization took approximately 5 minutes per image on an Intel Core i7-5960X CPU.

We adopted CMA-ES as the optimization algorithm because it gave the best results among the algorithms implemented in Optuna. The number of optimization steps was set to 2,000, and optimization was terminated if the evaluated value did not decrease by 0.001 or more in the last 500 steps.

The search ranges of parameters were set as follows. The camera translation vector was set as $t_{x},t_{y}\in[-0.1,0.1]$ and $t_{z}\in[-0.5,0.5]$, the camera rotation angles were $\theta_{\mathit{roll}},\theta_{\mathit{pitch}},\theta_{\mathit{yaw}}\in[-10\tcdegree,10\tcdegree]$, the vertical field-of-view angle was $\theta_{\mathit{fovy}}\in[-10\tcdegree,10\tcdegree]$. The coefficients $s_{w},s_{h}$ were initialized as 1, and the search range was set to $[0.1,2]$.

We conducted an ablation study of our method. Specifically, we compared three variants: optimized using (i) Adam as a gradient-based optimizer, (ii) CMA-ES as a black-box optimizer, and (iii) CMA-ES with scale adjustment (i.e. image aspect ratio optimization). As an evaluation metric, we adopted aesthetic scores estimated using VEN [21]. While VEN’s scores are unbounded, the larger score indicates more aesthetic. Table 1 shows a quantitative comparison of the three variants and the input images. We can see Adam only slightly improves the scores from the input images, but the margins are quite small. Contrarily, CMA-ES outperforms Adam by a large margin, and the scores are further improved by accounting for image size adjustment.

This trend can also be confirmed from the qualitative comparison shown in Figure 3. Adam improves the scores only slightly but the resultant images look almost the same as the input images, which indicates that Adam fell into local minima. Using CMA-ES, the scores are drastically improved and the 3D views are largely changed from the input images. Note that the camera’s viewing directions are clearly changed in the first and third rows of Figure 3 while preserving the 3D scene structure, which cannot be accomplished using traditional 2D cropping methods. The scores are further improved by accounting for the image scaling to adjust image aspect ratios. Also note that the aspect ratio might change significantly, for example, a vertical image became horizontal; see the bottom example in Figure 3. Hereafter we denote CMA-ES with image scaling as our method.

Next, we compared our method with existing 2D cropping methods. The compared methods are VPN [21], VEN [21], CGS [14], CAC [10], and UNIC [17]. As evaluation metrics, in addition to VEN [21], we also evaluated aesthetic scores using SAMPNet [24] and TANet [9].

Table 2 shows the quantitative evaluation result. Surprisingly, even though our method uses only VEN as an aesthetic score estimator during optimization, our results are consistently better than other methods when evaluated using not only VEN but also SAMPNet and TANet.

Figures 5, 4, and 6 show the results of qualitative comparison. The relative image sizes in each row are maintained from the orignal sizes using uniform scaling. The aesthetic scores under images are calculated using VEN. It can be observed that the existing methods attempted to extract the most salient objects within the rectangle regions, while our method strived to find views that capture the whole 3D scenes, presumably because VEN prefers such views. This strategic difference is evident in the decisive differences in the under-image aesthetic scores estimated using VEN.

To further validate our method, we conducted a user study with 15 test image sets (i.e., seven sets from Figure 5, six from Figure 4, and two from the first to the second rows of Figure 6). We compared the top-four methods, including the original images, in reference to Table 2, i.e., Input, VEN, UNIC, and Ours. 64 subjects were requested to pick the most aesthetically pleasing result for each test set, mentioning the importance of image composition. Consequently, the selection percentages are: Input 12.5%, VEN 24.3%, UNIC 17.3%, and Ours 45.9%, which means that ours outperforms other methods.

This user study also revealed an interesting tendency of subjects’ preferences. Overall, the subjects preferred images with a wider field of view to grasp the 3D scene structures, while they preferred images focusing on human subjects, if there were any in the scene. VEN also seemingly prefers a wider field of view, and thus, our optimized images have such views, resulting in the best selection percentage in this user study. This result stems from the characteristics of VEN, and a comprehensive investigation including other aesthetic score estimators is beyond the scope of our paper, which should be an exciting future work.

Currently, our method has the following limitations.