MM-PCQA: Multi-Modal Learning for No-reference Point Cloud Quality Assessment

In the early years of PCQA development, some simple point-based FR-PCQA methods are proposed by MPEG, such as p2point Mekuria et al. (2016a) and p2plane Tian et al. (2017). To further deal with colored point clouds, point-based PSNR-yuv Torlig et al. (2018) is carried out. Since the point-level difference is difficult to reflect complex distortions, many well-designed FR-PCQA metrics are proposed to employ structural features and have achieved considerable performance, which includes PCQM Meynet et al. (2020), GraphSIM Yang et al. (2020b), PointSSIM Alexiou and Ebrahimi (2020), etc.

To cover more range of practical applications and inspired by NR image quality assessment Zhang and Liu (2022); Zhang et al. (2022a, 2021b), some NR-PCQA methods have been proposed as well. Chetouani $et$ $al.$ Chetouani et al. (2021) extract patch-wise hand-crafted features and use classical CNN models for quality regression. PQA-net Liu et al. (2021b) utilizes multi-view projection for feature extraction. Zhang $et$ $al.$ Zhang et al. (2022c) use several statistical distributions to estimate quality-aware parameters from the geometry and color attributes’ distributions. Fan $et$ $al.$ Fan et al. (2022b) infer the visual quality of point clouds via the captured video sequences. Liu $et$ $al.$ Liu et al. (2022c) employ an end-to-end sparse CNN for quality prediction. Yang $et$ $al.$ Yang et al. (2022) further transfer the quality information from natural images to help understand the point cloud rendering images’ quality via domain adaptation. The mentioned methods are all based on single-modal information, thus failing to jointly incorporate the multi-modal quality information.

Many works Radford et al. (2021); Cheng et al. (2020) have proven that multi-modal learning is capable of strengthening feature representation by actively relating the compositional information across different modalities such as image, text, and audio. Afterwards, various works utilize both point clouds and image data to improve the understanding of 3D vision, which are mostly targeted at 3D detection. Namely, AVOD Ku et al. (2018) and MV3D Chen et al. (2020) make region proposals by mapping the LiDAR point clouds to bird’s eye view. Then Qi $et$ $al.$ Qi et al. (2018) and Wang $et$ $al.$ Wang and Jia (2019) attempt to localize the points by proposing 2D regions with 2D CNN object detector and then transforming the corresponding 2D pixels to 3D space. Pointpainting Vora et al. (2020) projects the points into the image segmentation mask and detects 3D objects via LiDAR-based detector. Similarly, PI-RCNN Xie et al. (2020) employs semantic feature maps and makes predictions through point-wise fusion. 3D-CVF Yoo et al. (2020) transfers the camera-view features using auto-calibrated projection and fuses the multi-modal features with gated fusion networks. More recently, some works Afham et al. (2022); Zhang et al. (2022b) try to make use of multi-modal information in a self-supervised manner and transfer the learned representation to the downstream tasks.

Suppose we have a colored point cloud $\mathbf{P}=\{g_{(i)},c_{(i)}\}_{i=1}^{N}$, where $g_{(i)}\in\mathbb{R}^{1\times 3}$ indicates the geometry coordinates, $c_{(i)}\in\mathbb{R}^{1\times 3}$ represents the attached RGB color information, and $N$ stands for the number of points. The point cloud modality $\mathbf{\hat{P}}$ is obtained by normalizing the original geometry coordinates and the image modality $\mathbf{I}$ is generated by rendering the colored point cloud $\mathbf{P}$ into 2D projections. Note that $\mathbf{\hat{P}}$ contains no color information.

It is natural to transfer mainstream 3D object detectors such as PointNet++ Qi et al. (2017) and DGCNN Wang et al. (2019) to visual quality representation of point clouds. However, different from the point clouds used for classification and segmentation, the high-quality point clouds usually contain large numbers of dense points, which makes it difficult to extract features directly from the source points unless utilizing down-sampling. Nevertheless, the common sampling methods are aimed at preserving semantic information whereas inevitably damaging the geometry patterns that are crucial for quality assessment. To avoid the geometry error caused by the down-sampling and preserve the smoothness and roughness of local patterns, we propose to gather several local sub-models from the point cloud for geometric structure feature representation.

Specifically, given a normalized point cloud $\mathbf{\hat{P}}$, we employ the farthest point sampling (FPS) to obtain $N_{\delta}$ anchor points $\{\delta_{m}\}_{m=1}^{N_{\delta}}$. For each anchor point, we utilize K nearest neighbor (KNN) algorithm to find $N_{s}$ neighboring points around the anchor point and form such points into a sub-model:

where $\rm{S}$ is the set of sub-models, $\boldsymbol{\rm{KNN}}(\cdot)$ denotes the KNN operation and $\delta_{m}$ indicates the $m$-th farthest sampling point. An illustration of the patch-up process is exhibited in Fig. 3. It can be seen that the local sub-models are capable to preserve the local patterns. Then a point cloud feature encoder $\theta_{P}(\cdot)$ is employed to map the obtained sub-models to quality-aware embedding space:

where $F_{P}^{l}\in\mathbb{R}^{1\times C_{P}}$ indicates the quality-aware embedding for the $l$-th sub-model $S_{l}$, $C_{P}$ represents the number of output channels of the point cloud encoder $\theta_{P}(\cdot)$, and $\widetilde{F}_{P}\in\mathbb{R}^{1\times C_{P}}$ is the pooled results after average fusion.

$N_{I}$ image projections are rendered from the distorted colored point clouds from a defined circular pathway with a fixed viewing distance to keep the texture consistent as shown in Fig. 4:

where the pathway is centered on the point cloud’s geometry center, $R$ indicates the fixed viewing distance and the $N_{I}$ image projections are captured with intervals of $\frac{2\pi}{N_{I}}$. The projections are rendered with the assistance of Open3D Zhou et al. (2018). Then we embed the rendered 2D images into quality-aware space with the 2D image encoder:

where $F_{I}^{t}\in\mathbb{R}^{1\times C_{I}}$ denotes the quality-aware embedding for the $t$-th image projection $I_{t}$, $C_{I}$ represents the number of output channels of the 2D image encoder $\theta_{I}(\cdot)$, and $\widetilde{F}_{I}\in\mathbb{R}^{1\times C_{I}}$ is the pooled results after average fusion.

As shown in Fig 1, the single modality may be incomplete to cover sufficient information for quality assessment, thus we propose a symmetric attention transformer block to investigate the interaction between the visual quality features gathered from the point cloud and image modalities. Given the intra-modal features $\widetilde{F}_{P}\in\mathbb{R}^{1\times C_{P}}$ and $\widetilde{F}_{I}\in\mathbb{R}^{1\times C_{I}}$ from the point clouds and images respectively, we adjust them to the same dimension with linear projection:

where $\hat{F}_{P}\in\mathbb{R}^{1\times C^{\prime}}$ and $\hat{F}_{I}\in\mathbb{R}^{1\times C^{\prime}}$ are adjusted features, $W_{P}$ and $W_{I}$ are learnable linear mappings, and $C^{\prime}$ is the number of adjusted channels. To further explore the discriminate components among the modalities, the multi-head attention module is utilized:

where $\Gamma(\cdot)$ indicates the multi-head attention operation, $\beta(\cdot)$ represents the attention function, $h_{\mu}$ is the $\mu$-th head, and $W$, $W_{Q}$, $W_{K}$, $W_{V}$ are learnable linear mappings. As illustrated in Fig. 5, both point-cloud-based and image-based quality-aware features participate in the attention learning of the other modality. The final quality embedding can be concatenated by the intra-modal features and the guided multi-modal features obtained by the symmetric cross-modality attention module:

where $\oplus(\cdot)$ indicates the concatenation operation, $\Psi(\cdot)$ stands for the symmetric cross-modality attention operation, and $\hat{F}_{Q}$ represents the final quality-aware features ($\Psi(\hat{F}_{P},\hat{F}_{I})\in\mathbb{R}^{1\times 2C^{\prime}}$, $\hat{F}_{Q}\in\mathbb{R}^{1\times 4C^{\prime}}$).

Following common practice, we simply use two-fold fully-connected layers to regress the quality features $\hat{F}_{Q}$ into predicted quality scores. For the quality assessment tasks, we not only focus on the accuracy of the predicted quality values but also lay importance on the quality rankings. Therefore, the loss function employed in this paper includes two parts: Mean Squared Error (MSE) and rank error. The MSE is utilized to keep the predicted values close to the quality labels, which can be derived as:

where $q_{\eta}$ is the predicted quality scores, $q_{\eta}^{\prime}$ is the quality labels of the point cloud, and $n$ is the size of the mini-batch. The rank loss can better assist the model to distinguish the quality difference when the point clouds have close quality labels. To this end, we use the differentiable rank function described in Sun et al. (2022) to approximate the rank loss:

where $i$ and $j$ are the corresponding indexes for two point clouds in a mini-batch and the rank loss can be derived as:

Then the loss function can be calculated as the weighted sum of MSE loss and rank loss:

where $\lambda_{1}$ and $\lambda_{2}$ are used to control the proportion of the MSE loss and the rank loss.

To test the performance of the proposed method, we employ the subjective point cloud assessment database (SJTU-PCQA) Yang et al. (2020a), the Waterloo point cloud assessment database (WPC) proposed by Liu et al. (2022a), and the WPC2.0 database Liu et al. (2021a) for validation. The SJTU-PCQA database includes 9 reference point clouds and each point cloud is corrupted with seven types of distortions (compression, color noise, geometric shift, down-sampling, and three distortion combinations) under six strengths, which generates 378 = 9$\times$7$\times$6 distorted point clouds in total. The WPC database contains 20 reference point clouds and augments each point cloud with four types of distortions (down-sampling, Gaussian white noise, Geometry-based Point Cloud Compression (G-PCC), and Video-based Point Cloud Compression (V-PCC)), which generates 740 = 20$\times$37 distorted point clouds. The WPC2.0 databases provides 16 reference point clouds and degradaes the point clouds with 25 V-PCC settings, which generates 400 = 16$\times$25 distorted point clouds.

The Adam optimizer Kingma and Ba (2015) is utilized with weight decay 1e-4, the initial learning rate is set as 5e-5, and the batch size is set as 8. The model is trained for 50 epochs by default. Specifically, We set the point cloud sub-model size $N_{s}$ as 2048, set the number of sub-models $N_{\delta}=6$, and set the number of image projections $N_{I}=4$. The projected images with the resolution of 1920$\times$1080$\times$3 are randomly cropped into image patches at the resolution of 224$\times$224$\times$3 as the inputs (the white background is removed from the projected images). The PointNet++ Qi et al. (2017) is utilized as the point cloud encoder and the ResNet50 He et al. (2016) is used as the image encoder, where the ResNet50 is initialized with the pre-trained model on the ImageNet database Deng et al. (2009). The multi-head attention module employs 8 heads and the feed-forward dimension is set as 2048. The weights $\lambda_{1}$ and $\lambda_{2}$ for $L_{MSE}$ and $L_{rank}$ are both set as 1.

Following the practices in Fan et al. (2022b), the k-fold cross validation strategy is employed for the experiment to accurately estimate the performance of the proposed method. Since the SJTU-PCQA, WPC, and WPC2.0 contain 9, 20, 16 groups of point clouds respectively, 9-fold, 5-fold, and 4-fold cross validation is selected for the three database to keep the train-test ratio around 8:2. The average performance is recorded as the final results. It’s worth noting that there is no content overlap between the training and testing sets. We strictly retrain the available baselines with the same database split set up to keep the comparison fair. What’s more, for the FR-PCQA methods that require no training, we simply validate them on the same testing sets and record the average performance.

14 state-of-the-art quality assessment methods are selected for comparison, which consist of 8 FR-PCQA methods and 6 NR-PCQA methods. The FR-PCQA methods include MSE-p2point (MSE-p2po) Mekuria et al. (2016a), Hausdorff-p2point (HD-p2po) Mekuria et al. (2016a), MSE-p2plane (MSE-p2pl) Tian et al. (2017), Hausdorff-p2plane (HD-p2pl) Tian et al. (2017), PSNR-yuv Torlig et al. (2018), PCQM Meynet et al. (2020), GraphSIM Yang et al. (2020b), and PointSSIM Alexiou and Ebrahimi (2020). The NR-PCQA methods include BRISQUE Mittal et al. (2012a), NIQE Mittal et al. (2012b), IL-NIQE Zhang et al. (2015), ResSCNN Liu et al. (2022c), PQA-net Liu et al. (2021b), and 3D-NSS Zhang et al. (2022c). Note that BRISQUE, NIQE, IL-NIQE are image-based quality assessment metrics and are validated on the same projected images. Furthermore, to deal with the scale differences between the predicted quality scores and the quality labels, a five-parameter logistic function is applied to map the predicted scores to subjective ratings, as suggested by Antkowiak et al. (2000).

Four mainstream evaluation criteria in the quality assessment field are utilized to compare the correlation between the predicted scores and MOSs, which include Spearman Rank Correlation Coefficient (SRCC), Kendall’s Rank Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC), Root Mean Squared Error (RMSE). An excellent quality assessment model should obtain values of SRCC, KRCC, PLCC close to 1 and RMSE to 0.

As described above, MM-PCQA jointly employs the features from the point cloud and image modalities and we hold the hypothesis that multi-modal learning helps the model to gain better quality representation than single-modality. Therefore, in this section, we conduct ablation studies to validate our hypothesis. The performance results are presented in Table 2. The performance of both single-modal-based model is inferior to the multi-modal-based model, which suggests that both point cloud and image features make contributions to the final results. After using SCMA module, the performance is further boosted, which validates the effectiveness of cross-modality attention.

To prove that the patch-up strategy is more suitable for PCQA, we present the performance of using farthest point sampling (FPS) strategy as well. To make the comparison even, 12,288 = 6$\times$2048 points (the same number of points as contained in the default 6 sub-models) are sampled for each point cloud and the results are shown in Table 3. It can be seen that the patch-up strategy greatly improves the performance when only point cloud features are used. The reason is that the sampled points are not able to preserve the local patterns that are vital for quality assessment. Moreover, when the point cloud contains more points, the sampled points may not even maintain the main shape of the object unless greatly increasing the number of sampling points.

The cross-database evaluation is further conducted to test the generalization ability of the proposed MM-PCQA and the experimental results are exhibited in Table 4. Since the WPC database is the largest in scale (740), we mainly train the models on the WPC database and conduct the validation on the SJTU-PCQA (378) and WPC2.0 (400) databases. From the table, we can find that the proposed MM-PCQA better generalizes learned feature representation from the WPC database and achieves much better performance than the most competitive NR-PCQA methods. Furthermore, the WPC$\rightarrow$WPC2.0 MM-PCQA is even superior to all the competitors directly trained on the WPC2.0 database.

We try to investigate the contributions of the point cloud and image branch by varying the number of the 3D sub-models and input 2D projections. The performance results are exhibited in Table 5. We can see that employing 6 sub-models and 4 projections yields the best performance on major aspects for the point cloud and image branch respectively. With the number increasing, the features extracted from sub-models and projections may get redundant, thus resulting in the performance drop.

In this section, we present the performance of different feature encoders. The popular 2D image backbones VGG16 Simonyan and Zisserman (2014), MobileNetV2 Sandler et al. (2018) and ResNet50 He et al. (2016) are used for demonstration while the mainstream point cloud backbones PointNet++ Qi et al. (2017) and DGCNN Wang et al. (2019) are also included. The results are shown in Table 6. It can be found that the proposed PointNet++ and ResNet50 combination is superior to other encoder combinations. Additionally, the performance of different backbones are still competitive compared with other NR-PCQA methods, which further confirms the effectiveness of the proposed framework.