Asynchronous Feedback Network for Perceptual Point Cloud Quality Assessment

The aforementioned analysis reveals that human perception is a dual-stream asynchronous process, and global feature are usually first perceived to guide the extraction of local feature. On the basis of the philosophy, as illustrated in Fig. 2, the framework of our model is divided into three stages: i) First, global feature are extracted based on the projected three types of multi-view point cloud images (i.e., texture, depth, and occupancy). We divide the texture and depth images into patches and feed them into one vision transformer (ViT) [38] pre-trained on ImageNet [39]. The attention maps of the class token (a learnable embedding for semantic prediction) in the ViT are merged into the global feature. ii) Second, we utilize the attention maps to guide the extraction of local feature. We use the attention maps to generate a mask map and several regional filters, the former highlighting multiple semantic regions. Based on the generated mask map and regional filters, we perform dynamic convolution for different regions of the original projected images and obtain the compact local feature, which are then fused with the global feature. iii) Third, we perform a coarse-to-fine quality prediction. The global and fused features are both supervised by the overall quality score, and a branch-wise quality ranking loss is applied to guarantee a progressive quality prediction.

Instead of inferring point cloud quality directly in 3D space, we project point clouds into 2D images to better leverage mature 2D pre-trained models targeted for semantic-related tasks (e.g., classification). Given a point cloud $X$, we render it into multi-view images from six perpendicular perspectives (that is, along the positive and negative directions of the x,y,z axes). To reduce information loss during the projection process, we obtain three different types of image to represent $X$: texture images, depth images, and occupancy images. More specifically, we use the PointsRenderer function in the Pytorch3D [40] library for point cloud projection. The renderer uses an orthographic camera, and applies alpha compositing to generate the pixel values (please refer to [41] for more details). The acquired former two types of image reflect the original color and geometry information and are normalized to the range $[0,1]$. The occupancy images are binary images locating the point cloud contents, where “1” indicates the point cloud contents while “0” indicates background areas. We separately denote the three types of multi-view images of $X$ as $\{X_{t}^{i}\in\mathbb{R}^{H\times W\times 3}|_{i=1}^{6}\}$, $\{X_{d}^{i}\in\mathbb{R}^{H\times W}|_{i=1}^{6}\}$ and $\{{X}_{o}^{i}\in\mathbb{R}^{H\times W}|_{i=1}^{6}\}$, where $i$ denotes the $i$-th viewpoint.

The vision transformer (ViT) has been shown to be capable of building long-term dependencies and prioritizing low-frequency information more than CNN [42, 43], making it a good choice for global feature extraction. To make the projected images compatible with the ViT, for ${X}_{t}$ and ${X}_{d}$ from an arbitrary perspective, we first partition them into non-overlapping $P\times P$ patches.

Encoding by Vision Transformer. Given a texture patch ${p}_{t}$ and its corresponding depth patch ${p}_{d}$, we separately map two patches into two vector representations and simply fuse them with position embedding (denoted by $\mathrm{PE}$), i.e.,

where ${x}\in\mathbb{R}^{D}$ is the token embedding; $\mathrm{PatchEmbed}_{t(d)}(\cdot)$ are two independent embedding operations.

Next, we follow [38] to prepend a learnable embedding ${x}_{c}$ named class token to the sequence of embedded patches. The token sequence can be represented as ${z}_{0}=[{x}_{c};{x}_{1};{x}_{2};\cdots;{x}_{N}]\in\mathbb{R}^{(N+1)\times D}$, where $N=HW/P^{2}$ denotes the patch number. We then input ${z}_{0}$ into the ViT, which consists of $L$ consecutive identical transformer blocks and is initialized with the optimized weights on ImageNet. Taking the $l$-th block as an example, it takes the output ${z}_{l-1}$ of the previous $(l-1)$-th block as input to compute its output ${z}_{l}$.

Attention Map Extraction. The ViT effectively models the relationship between patches using multi-head self-attention operation $\mathrm{MSA}(\cdot)$ [44]. For the $j$-th head of $\mathrm{MSA}(\cdot)$, the computational process can formulated as follows:

where $\mathrm{SA_{j}(\cdot)}$ represent the $j$-th self-attention operation, $j\in[1,N_{h}]$; $\mathrm{LN(\cdot)}$ denote the layer normalization [45]; $\Gamma_{j}$ denotes the projection matrix; $A_{j}$ denotes the attention matrix. According to Eq. (2), we can see that each element of the attention matrix $A_{j}$ indicates the pairwise correlation between two patch representations in ${z}_{l-1}$. Furthermore, the class token collects information from all patches via the self-attention operation. In traditional classification tasks, the output of the class token (i.e., $z_{L}[:,1]$) will be transformed into a semantic prediction via a small MLP [38]. Therefore, the correlation between the class token and an arbitrary patch representation can effectively reflect the relevance between the patch and semantics, which facilitates the generation of needed global feature. We define the attention maps $A_{c}$ of the class token as:

where $\oplus$ denotes the concatenation operation. To better illustrate the semantic sensitivity of the attention maps, we visualize $A_{c}$ in Fig. 3 (b), from which we can observe that the attention maps accurately emphasize the content information in the projected images.

Global feature Generation. After extracting the attention maps of the class token, we further transform $A_{c}$ in the last transformer block into global feature. Specifically, for the $i$-th viewpoint, the attention maps are processed by a $1\times 1$ convolution and an adaptive average pooling operation to generate a vector representation $f_{g}^{i}$. Given the different proportions of content among perspectives, we follow [46] to adopt a simple occupancy-weighted fusion strategy, that is,

where $w_{i}$ denotes the occupancy ratio, that is, the proportion of “1” in the occupancy image $X_{o}^{i}$. We take $f_{g}\in\mathbb{R}^{D_{o}}$ as the final global feature.

As mentioned in Section I, human observers can easily distinguish different semantic regions with a glance, and feedback connections add details for each region with scrutiny. Therefore, instead of learning only one convolution kernel for all positions, we intend to adaptively generate filters for different regions based on the attention maps. Considering that the same semantic regions may share among different viewpoints, we first concatenate the texture and depth images of $X$ and compose the multi-view images into a stitched image denoted as $S\in\mathbb{R}^{2H\times 3W\times 4}$, which corresponds to a stitched occupancy image denoted as $S_{o}\in\mathbb{R}^{2H\times 3W}$.

Region-aware Feedback. In order to dynamically process different semantic regions, we choose DRConv [22] to take advantage of its idea of producing region-wise filters. Unlike the original DRConv that learns different filters and region divisions directly from the input images, we learn the two terms from the attention maps. This type of design can better exploit the guiding role of global feature.

Specifically, the attention maps of different viewpoints are first stitched as $A_{s}\in\mathbb{R}^{\frac{2H}{P}\times\frac{3W}{P}\times N_{h}}$. Considering that the occupancy images directly distinguish the blank backgrounds using “0”, it is reasonable to merge the occupancy information into the attention maps to achieve better region division. For this purpose, the stitched occupancy image $S_{o}$ is first resized to the size of $\frac{2H}{P}\times\frac{3W}{P}$, and then multiply with each channel of $A_{s}$, i.e.,

where $\odot$ is element-wise multiplication.

After enhancing the attention maps with occupancy information, we feed $\tilde{A}_{s}$ into a dual-stream embedder, which includes two modules: mask prediction module and filter generation module (see Fig. 4 (a) for the explicit structure). The former module outputs a guided mask $M=\{M_{1},\cdots,M_{n}\}\in\mathbb{R}^{2H\times 3W}$ representing the region division, in which pixel values vary from 0 to $n-1$ and indicate $n$ different semantic regions. The latter module generates $n$ groups of $k\times k$ filter as $W=\{W_{1},\cdots,W_{n}\}$, where the filter $W_{t}$ corresponds to the region $M_{t}$, $t\in[1,n]$. The feature sizes of the feedback module are listed in Table I. Furthermore, we illustrate the generated masks used in Fig. 3 (c). From the figure, we can observe that the masks can distinguish different semantic regions (e.g., head, background), which is consistent with the HVS.

Based on $M$ and $W$, we perform the region-aware convolution for the stitched image $S$, which can be formulated as:

where $F_{u,v}$ denote the feature output in the position $(u,v)$; $\otimes$ is the 2D convolution operation. By applying a specific convolution kernel on a specific region, we enable the network to differentiate regions of different semantics, which promotes the extraction of local feature.

Local Feature Generation. After performing a region-aware convolution for the stitched image $S$, we propose to extract local feature through a shallow convolutional network, seeing Fig. 4. As illustrated in the figure, we use filters with a very small receptive field ($3\times 3$ and $1\times 1$), which makes that the extracted features are localized and usually refer to some image details (e.g., edges, contours). We show some feature maps of the local branch in the Fig. 5. The feature maps mainly concern the contour details of point clouds, which is complementary to the semantic information indicated by the attention maps. Finally, we obtain the local feature indicated by $f_{l}\in\mathbb{R}^{D_{o}}$.

After obtaining the global feature $f_{g}$ and the local feature $f_{l}$, to increase the feature discrepancy, we define the feature disentangling loss as follows:

According to Eq. (7), when the cosine similarity between $f_{g}$ and $f_{l}$ becomes 0, it indicates that there is no linear correlation between the two features. A margin was given to the loss function because being exactly opposite could harm the representation ability [47].

Then we intend to make a coarse-to-fine quality prediction. Specifically, we define two quality regression heads for two branches:

where $\mathrm{FC}(\cdot)$ is a full-connection layer; $\hat{q}_{c}$ and $\hat{q}_{f}$ denote the coarse-grained and fine-grained predicted quality scores respectively. Denoting the subjective mean opinion score (MOS) as $q$, both two predictions are supervised by the regression loss as:

where the superscript “$b$” denotes the $b$-th samples in a mini-batch with the size of $B$. To achieve a progressive quality prediction, we further propose a branch-wise quality ranking loss as:

where $\mathrm{SROCC}(\cdot)$ denotes the Spearman rank-order correlation coefficient between the predictions and the ground truths in the mini-batch. Note that we refer to [48] to create the differentiable implementation of the ranking metric. Eq. (10) enforces $\mathrm{SROCC}(\hat{q}_{c},q)<\mathrm{SROCC}(\hat{q}_{f},q)$ during training. Such a design can enable networks to take the coarse prediction from the global feature and gradually approach the finer-grained prediction by supplementing the local feature.

Finally, the overall loss function for training is defined as

where $\lambda_{1}$ and $\lambda_{2}$ are the weighting factors.

Datasets. Our experiments are based on three commonly used PCQA datasets, including LS-PCQA Part I[13], SJTU-PCQA [23], and WPC [24]. LS-PCQA Part I consists of 930 annotated point clouds impaired by 31 types of distortion, which are generated from 85 references. SJTU-PCQA includes 9 reference point clouds and 378 distorted samples impaired with 7 types of distortions (e.g., color noise, downsampling) under 6 levels. WPC contains 20 reference point clouds and 740 distorted samples disturbed by 5 types of distortions (e.g., compression, gaussian noise).

Evaluation Metrics. Three widely adopted evaluation metrics are employed to quantify the level of agreement between predicted quality scores and the MOS: Spearman rank order correlation coefficient (SROCC), Pearson linear correlation coefficient (PLCC), and root mean square error (RMSE). To ensure consistency between predicted scores and MOSs, a nonlinear Logistic-4 regression is applied to map the predicted scores to the same dynamic range, following the recommendations suggested by the Video Quality Experts Group (VQEG) [49], i.e.,

where $\beta_{1}$, $\beta_{2}$, $\beta_{3}$, and $\beta_{4}$ are the parameters fitted by minimizing the sum of squared errors.

We implement our model by PyTorch and conduct training and testing on the NVIDIA 3090 GPUs. All point clouds are rendered into three types of projected images with a spatial resolution of $512\times 512$ by PyTorch3D [40]. The used vision transformer is ViT-B [38].

Dataset Split. Considering the limited scale of the data set, the k-fold cross-validation strategy is used for the experiment to accurately estimate the performance of the proposed method. Following [4], 9-fold and 5-fold cross-validation is selected for SJTU-PCQA (9 references) and WPC (20 references). For LS-PCQA (85 references), we apply a 5-fold cross-validation. Note that there is no content overlap between the training and testing sets for the three databases. Specifically, the ratios of content in the training set and test set are 8:1, 16:4, and 68:17 for the three databases. For each fold, the performance on the test set with minimal training loss is recorded, and the average performance across all folds is recorded as the final result.

Training Strategy. All databases are trained for 50 epochs with a batch size of 8. During the training process, all training images are deployed with the traditional data enhancement strategy, i.e., randomly cropping the images to $224\times 224$ size such as [33, 4]. During the testing stage, following [33], 10 patches with $224\times 224$ size are randomly sampled and their corresponding prediction scores are average pooled to obtain the final quality score. We use the Adam [58] optimizer with weight decay $5e-4$. The learning rate is set separately as $2e-5$ and $2e-4$ for the pre-trained model and other parts, and is reduced by a rate of 0.9 every 5 epochs.

Network Details. To make the projected images compatible with ViT-B, the input patch size of the global branch is $16\times 16$. The MSA head number is set by default as $N_{h}=12$. The number of regions in the guided mask is set to $n=8$ and the kernel size of region-wise filters is set to $k=3$. The global and local feature both have a dimension of $D_{o}=256$. The weighting factors in the loss function are simply set to $\lambda_{1}=\lambda_{2}=1$.

Overall Performance Comparison. Table II lists the experimental results on META-3D . The proposed method, AFQ-Net, is compared with 11 FR-PCQA metrics, and 7 NR-PCQA metrics. we run all published codes to produce the results. We strictly retrain the available baselines with the same database split set up to keep the comparison fair. In addition, for the FR-PCQA methods that require no training, we simply validate them on the same testing sets and record the average performance.

For each database, the top two performance results for each evaluation criterion are highlighted in boldface and underline. We can see that the proposed AFQ-Net exhibits outstanding performance across all three datasets. Especially, the proposed AFQ-Net presents remarkable performance on the LS-PCQA, a database with wide diversity in terms of both content and distortion type, which demonstrates the adaptability and robustness of our model. In comparison, the performance of the existing FR and NR methods varies significantly across different datasets. For example, GMS-3DQA demonstrates strong performance on LS-PCQA but exhibits poor performance on WPC; 3DTA works well on SJTU-PCQA but presents inferior performance on LS-PCQA. IT-PCQA performs poorly because it is based on domain adaptation, in which the labels of point clouds are not utilized in the training. Meanwhile, it should be noted that our method presents competitive performance compared to the FR-PCQA methods despite the inaccessibility of the reference information.

To further validate the model performance in compression scenarios, we test the model performance on two PCQA databases that consist only of compressed samples, i.e., the M-PCCD database (232 distorted samples corresponding to 8 references) [59] and the BASICS database (1494 distorted samples corresponding to 75 references) [60]. 4-fold and 5-fold cross-validations are applied for the two databases, respectively. We report the performance of different NR-PCQA metrics on M-PCCD and BASICS in Tab III. According to the table, we can see that AFQ-Net provides the best SROCC on M-PCCD and the best PLCC and RMSE on BASICS, which demonstrates that the proposed method is sensitive to compression distortions. MM-PCQA and 3DTA both presents inferior performance on one database (MM-PCQA on BASICS and 3DTA on M-PCCD). GMS-3DQA works well, which is also based on 2D projection. Based on the results, it seems that metrics using 3D information present unstable performance on different databases. This may be because those metrics only utilize some local patches for quality prediction, which lacks global perception and is easily affected by point density. However, for those metrics that only leverage 2D projection, how to mimic different rendering strategies for different databases has not yet been explored in depth.

We also show some examples of distorted point clouds with the subjective MOS, the predicted score of the proposed AFQ-Net and other NR-PCQA models in Fig. 6, where two point clouds are impaired by G-PCC trisoup compression and additive noise. It can be seen that the predicted score provided by AFQ-Net aligns well with subjective perception. In comparison, other models are not sensitive to quality changes because their predicted scores for two point clouds are relatively close.

Qualitative Analysis. To demonstrate that our model can attentively capture the critical features related to distortions, we also present a qualitative analysis using a prevalent tool for network explanation, that is, Grad-CAM++ [61]. As illustrated in Fig. 7, subfigure (a) presents three samples with the same content but impaired by three different distortions (i.e., Rayleigh noise, V-PCC compression, and G-PCC compression), and subfigure (b) presents their Grad-CAM++ maps that reflect what makes the network perform decision. Note that the extracted activation maps correspond to the last convonlutional layer on the local branch (i.e., the $1\times 1$ Conv in Fig. 4 (b)). From the figure, we have the following observations: i) the activation map associated with Rayleigh noise distortion mostly highlights these flat regions. This is mainly because Rayleigh noise is located at isolated points. According to texture masking effect [62], noise in complex texture regions (e.g., orange box) is generally harder to perceive than noise in homogeneous regions (e.g., blue box). ii) For two types of compression distortion, attribute quantization in V-PCC leads to abnormal color mutations, whereas Octree decomposition in G-PCC causes a sparse point distribution. Consequently, the activation maps associated with V-PCC and G-PCC emphasize the whole point cloud because these deformations are severe and not localized. Based on these instances, we can see that the proposed AFQ-Net can effectively capture and distinguish various distortions.

Cross-Dataset Evaluation. The cross-dataset evaluation is conducted to test the generalizability of the NR-PCQA methods when encountering various data distributions. In Table IV, we first train the compared models on one database and test the trained model on another database, and then swap the training set and the test set. Note that we use the whole database for training or testing. For each validation, we record the result with minimal training loss and report the results in Table IV. From the table, we can see that: i) AFQ-Net achieves good performance when trained on LS-PCQA, which shows the generalizability of our model when rich content categories are provided. ii) Almost all methods (including AFQ-Net) show poor performance when trained on SJTU-PCQA and tested on LS-PCQA and WPC (denoted by SJTU2LS and SJTU2WPC validation). This is largely due to the limited and extremely biased content in SJTU-PCQA (5 out of 9 contents are about the human body). iii) AFQ-Net trained on WPC shows opposite trends when validated on SJTU-PCQA and LS-PCQA. More specifically, AFQ-Net performs the best when validated on LS-PCQA but presents relatively inferior performance when validated on SJTU-PCQA. This may be due to the large difference in content categories between SJTU-PCQA and WPC (human body vs. object). Notably, some recent methods demonstrate superior cross-dataset performance on several validations. For instance, MM-PCQA and 3DTA perform well on the WPC2SJTU validation, likely due to their use of 3D information as input, which enables them to capture common density variations in SJTU-PCQA more effectively. GMS-3DQA achieves the best results on the SJTU2WPC validation, potentially because of its mini-patch sampling strategy. The mini-patch map in GMS-3DQA preserves the distortion patterns of point clouds while blurring content information, which appears advantageous when there are substantial content differences between databases. To enhance the performance of AFQ-Net, a promising approach would be to replace the stitched image in the local branch with the mini-patch map, thus improving its capacity to evaluate local distortions. Additionally, directly applying the asynchronous feedback strategy for 3D data is also a practical and intriguing avenue.

Ablation Study for the Key Modules. To verify the effectiveness of the key modules in the AFQ-Net, we further investigate the contribution of different components and report the results in Table V. In the table, the index (1) and (2) denote using only global branch or local branch for quality prediction respectively. (3) indicates one dual-branch network without feedback connection (i.e., replacing the DRConv in Fig. 4 (b) with $3\times 3$ convolution), while (4) represents the full model. From the table, we have the following observations: i) Seeing (1) and (2), we see that using only the global branch performs better than using only the local branch structure. This is mainly because the parameter size of the global branch is significantly larger than that of the local branch; consequently, the global branch has stronger fitting ability. Meanwhile, the pre-trainning on ImageNet inject the global branch some quality-related prior knowledge, which boosts its performance. ii) Comparing (1) or (2) with (3), the dual-branch structure performs significantly better than the single-branch structure, demonstrating the complementarity of the two features; iii) Seeing (3) and (4), the feedback module further brings a noticeable gain, which verifies the effectiveness of the feedback mechanism in visual quality assessment.

Ablation Study for the Region Count. We test the performance of AFQ-Net under different $n$ values to investigate the influence of the division granularity and report the results in Fig. 8. We see that SJTU-PCQA, WPC and LS-PCQA respectively provide the best performance with $n=6,8,$ and $10$. This may be due to the different amount of content in the three databases. SJTU-PCQA has limited and extremely biased content, so few divisions can distinguish different semantic regions while too many divisions cause over-segmentation for the same region. In comparison, WPC and LS-PCQA have more references (20 for WPC and 85 for LS-PCQA) and richer content including different object categories, for which cases large $n$ values can benefit the model performance.

Ablation Study for the Kernel Size. We test the performance of AFQ-Net under different $k$ values to investigate the influence of the kernel size of region-wise filters and report the results in Fig. 9 (a) and (b). We see that when the kernel size is too large (e.g., $k=7$), the model performance will decrease. Especially, $7\times 7$ kernel causes inferior results on LS-PCQA. Further looking at this case, we find that larger kernel size in DRConv will generate larger element values in the local feature, which will make the local feature contribute more to the prediction, thus affecting the convergence speed of the network. More specifically, given the local feature $f_{l}$, we define its average absolute value of as $\mu_{l}=\mathrm{mean}(\mathrm{abs}(f_{l}))$.We have illustrated the SROCC (on the training set) trend and the $\mu_{l}$ trend versus the training epoch for $3\times 3$ and $7\times 7$ kernel in Fig. 10 (a) and (b). From Fig. 10 (b), we can see the $\mu_{l}$ of $7\times 7$ kernel is significantly larger than that of $3\times 3$ kernel; according to Fig. 10 (a), the corresponding SROCC on the training set converges more slowly. In fact, DRConv tends to generate element values in the range of $[0,1]$ for convolution kernels, therefore causing larger results with increasing kernel size. Meanwhile, the complexity of LS-PCQA make the network harder to converge compared to WPC and SJTU-PCQA. A reasonable solution is to add a normalization factor for different kernel size, such as $\frac{1}{9}$ for $3\times 3$ kernel and $\frac{1}{49}$ for $7\times 7$ kernel. We report the SROCC and the $\mu_{l}$ trend of two normalized kernels in Fig. 10 (c) and (d). We can see the normalized $3\times 3$ and $7\times 7$ kernels present similar trends in both two sub-figures. Meanwhile, the final [PLCC,SROCC,RMSE] criteria for the two normalized kernels are $[0.6826,0.6706,0.5858]$ and $[0.6755,0.6664,0.5962]$ respectively, which demonstrates that the normalization can effectively eliminate the performance drop caused by large kernels.

Ablation Study for the Input Image Types. Three types of image are generated during the preprocessing stage. The depth images serve as a supplement to texture images in the network input stage, while the occupancy images help perform multi-view fusion and region-aware feedback. We conduct experiments to validate the effectiveness of the three types of input and list the results in Table VI. In the table, index (1) and (2) represent using only texture or depth images for quality prediction respectively. (3), (4), and (5) denote pairwise combinations of different image types, including “texture+depth”, “depth+occupancy”, and “texture+occupancy”, while (6) indicates the full model. Note that the framework using only occupancy images is infeasible because they are only used to generate some weights and masks. From the table, we can observe: i) based on (1) and (2), model using texture images performs far better than that using depth images. It is reasonable because texture images generally contain visual information than depth images. ii) Comparing (1) with (3), introducing the depth images brings a marginal gain and even impairs the model performance. This is due to the fact that geometry distortions can also be reflected in texture images, therefore reducing the importance of depth information. Especially, most distortion types in LS-PCQA do not cause any geometry deformation, which result in relatively inferior performance when injecting depth information. iii) Seeing (1) and (5), (3) and (6), the introduction of occupancy images leads to noticeable improvements for the three criteria, demonstrating the importance of occupancy information in our model.

Ablation Study for the Loss Functions. The proposed network is trained using the regression loss $L_{reg}$, the feature disentangling loss $L_{dis}$, and the quality ranking loss $L_{rank}$. We evaluate the effect of each loss function and report the results in Table VII. In the table, index (1), (2) and (3) represent using single loss term for quality prediction respectively. (3), (4), and (5) denote pairwise combinations of different loss terms, including $L_{reg}+L_{dis}$, $L_{dis}+L_{rank}$, and $L_{reg}+L_{rank}$, while (6) indicates the full loss. From the table, it is observed: i) seeing (1), (4), and (5), both $L_{dis}$ and $L_{rank}$ benefit the model performance, demonstrating the effectiveness of the two loss functions. ii) According to (6) and (7), the full loss provides noticeable gain over the combination of $L_{reg}+L_{rank}$ on LS-PCQA but presents marginal gain on WPC. The reason may be that $L_{rank}$ also achieves feature disentanglement to some extent, which weakens the effect of $L_{dis}$. Considering $L_{dis}$ only measures the linear similarity between global and local features, one possible avenue for further improvement is to minimize non-linear dependence between two features, such as mutual information.