Perceptual Quality Assessment for Digital Human Heads

Sensing, Modeling, and driving digital humans have been popular research topics in computer vision and computer graphics. Namely, 3D human pose and shape estimation [7, 8] are of great significance for the representation of digital human models. Digital human animation generation [9, 10] aims to drive digital human motion activities with text or audio. However, little attention is paid to the visual quality assessment of digital humans.

Currently, 3D quality assessment (3D-QA) mainly focuses on point cloud quality assessment (PCQA) and mesh quality assessment (MQA). To cope with the challenge of visual quality assessment of 3D models, substantial efforts have been made to carry out 3D quality assessment databases [2, 3, 4, 5, 6] and the detailed comparison with the proposed database is listed in Table 1, from which we can find that the proposed DHHQA database is the first large-scale database specially designed for DHHs.

The objective quality assessment methods for 3D-QA can be generally categorized into two types: model-based methods [3, 4, 5, 11, 12] and projection-based methods [2, 6, 13, 14, 15]. Model-based methods extract features directly from 3D models while projection-based methods infer the visual quality of 3D models from the rendered projections. Although the model-based methods are invariant to viewpoints, it is difficult to efficiently extract quality-aware features from 3D models since high-quality 3D models usually contain large numbers of dense points/vertices. While the projection-based methods can make use of the mature image/video quality assessment (IQA/VQA) models, therefore gaining better performance.

To gather the source models of DHHs, we invite 255 human subjects (aged from 12 to 60) to participate in the scanning experiment with the Bellus3D ^††https://www.bellus3d.com app. Finally, a total of 55 DHHs are manually chosen as the source DHH models. Samples of the selected DHHs are exhibited in Fig. 1. Then 7 types of common distortions are applied to degrade the reference DHH models: (a) Surface simplification (SS): The algorithm proposed in [16] is employed to simplify the DHH models and the simplification rate is set as {0.4, 0.2, 0.1, 0.05}; (b) Position Compression (PC): The Draco ^††https://github.com/google/draco library is applied to quantize the position attributes of the DHH models with compression parameter qp set as {6, 7, 8, 9}; (c) UV Map Compression (UMC): The Draco library is applied to quantize the texture coordinate attributes with the compression parameter qt set as {7,8,9,10}; (d) Texture Down-sampling (TD): The reference texture maps (4096$\times$4096) are down-sampled with resolutions of {2048$\times$2048, 1024$\times$1024, 512$\times$512, 256$\times$256}; (e) Texture Compression (TC): The JPEG compression is used to compress the texture maps with quality levels set as {3, 10, 15, 20}; (f) Geometry Noise (GN): Gaussian noise is introduced to the vertices of the DHH models with $\sigma_{g}$ set as {0.05, 0.1, 0.15, 0.2}; (g) Color Noise (CN): Gaussian noise is added to the texture maps with $\sigma_{c}$ set as {20, 40, 60, 80}. To sum up, 1,540 = 55$\times$7$\times$4 distorted DHH models are generated and the degradation levels are manually defined to cover the most quality range.

Following the recommendation of ITU-R BT.500-13 [17], we conduct the subjective quality assessment experiment in a well-controlled laboratory environment. All the DHH models are rendered into projections from the front and left side viewpoint and the reference as well as distorted projections are at the same time for evaluation. The projections are shown in random order with an interface designed by Python Tkinter on an iMac monitor which supports the resolution up to 4096 $\times$ 2304, the screenshot of which is illustrated in Fig. 3. 22 males and 18 females are invited to participate in the subjective experiment. The whole experiment is divided into 11 sessions. Each session contains 140 distorted DHH models and is attended by at least 20 participants. In all, more than 30,800 = 1,540$\times$20 subjective ratings are collected.

After the subjective experiment, we calculate the z-scores from the raw ratings as follows:

where $\mu_{i}=\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}r_{ij}$, $\sigma_{i}=\sqrt{\frac{1}{N_{i}-1}\sum_{j=1}^{N_{i}}\left(r_{ij}-\mu_{i}\right)}$, and $N_{i}$ is the number of images judged by subject $i$. The ratings from unreliable subjects according to ITU-R BT.500-13 [17]. The corresponding z-scores are linearly rescaled to $[0,100]$ and the mean opinion scores (MOSs) are computed by averaging the rescaled z-scores. The MOS distribution is exhibited in Fig. 4, from which we can see that the quality ratings cover most of the quality range.

Following the common process of mapping the reference and distorted projections into quality embeddings with DNNs, the pretrained Swin Transformer tiny (ST-t) [18] is employed as the feature extraction backbone since it takes up less training and inference resources. Considering the visual information is perceived hierarchically from simple low-level features (e.g., noise and texture) to complex high-level features (e.g., semantic information), we employ the hierarchical structure for feature extraction:

where $S_{k}(x)$ represents the features from the $k$-th stage, ${\rm{AP}}(\cdot)$ stands for the average pooling operation, $\alpha_{k}(x)$ denotes the pooled results from the $k$-th stage, and $\oplus$ indicates the concatenation operation. Given the input reference and distorted front projections $P_{r}$ and $P_{d}$, the quality-aware features can be obtained as:

where $En(\cdot)$ is the hierarchical ST-t encoder, $F_{r}$ and $F_{d}$ represent the quality-aware embeddings for the input reference and distorted front projections.

To actively relate the quality-aware information between the reference and distorted embeddings, we utilize the multi-head attention operation for feature fusion:

where $\Gamma(\cdot)$ indicates the multi-head attention operation where $F_{r}$ is used to guide the attention learning of $F_{d}$, $F_{a}$ is the fused quality-aware embedding, and $F_{q}$ is the final quality embedding.

With the final quality embedding $F_{q}$, two-stage fully-connected (FC) layers are employed for feature regression:

where $Q^{\prime}$ represents the predicted quality scores and $\mathbf{FC}(\cdot)$ stands for the fully-connected layers. Moreover, the L1 loss is utilized as the loss function:

where $n$ is the size of the mini-batch, $Q_{\eta}$ and $Q^{\prime}_{\eta}$ are quality labels and predicted quality scores respectively.

The ST-t backbone is fixed with weights pretrained on the ImageNet [19] and only the feature fusion and regression modules are trained. The Adam optimizer [20] is used with the initial learning rate set as 1e-4 and the number of training epochs is set as 50. The batch size is set as 32. The 1080$\times$1920 front projections are resized into the resolution of 256$\times$456 and patches with the resolution of 224$\times$224 as the input. We conduct the 5-fold cross validation on the DHHQA database and the average results of the 5 folds are recorded as the final performance. It’s worth mentioning that there is no content overlap between the training and testing sets. Several mainstream FR-IQA methods are validated for comparison as well, which include PSNR, SSIM [21], MS-SSIM [22], GMSD [23], LPIPS [24], and PieAPP [25]. These methods are validated using the same front projections on the same testing sets as the proposed method and the average results are recorded as the final performance. We also include the performance of some classic FR point cloud quality assessment (PCQA) methods such as p2point[26], p2plane[27] and PSNR-yuv[28] through converting the DHHs from textured meshes to point clouds. Additionally, all the quality scores predicted by the IQA methods are processed with five-parameter logistic regression to deal with the scale difference.

The performance results are shown in Table 2, from which we can several inspections: (a) The proposed method achieves the higher performance among all the competitors and is about 0.03 ahead of the second place competitor LPIPS (0.9286 vs. 0.8935) in terms of SRCC, which shows the effectiveness of the proposed method for predicting the visual quality of DHH models; (b) Comparing the performance of MS-SSIM and SSIM, we can conclude that the multi-scale features can greatly help boost the performance since the MS-SSIM surpasses SSIM by almost 0.12 in terms of SRCC, which also reflects the rationality of the proposed hierarchical ST-t structure; (c) The learning-based methods (LPIPS, PieAPP, and proposed method) are all superior to the handcrafted-based methods. It can be explained that the handcrafted-based IQA methods are developed mainly for natural scene images (NSIs). However, the prior knowledge for perceptual quality differs from NSIs to DHH projections since the projections are artificially rendered with computers rather than captured with cameras. By actively learning the prior knowledge from the DHH projections, the learning-based methods can yield better performance consequently.